Repair/Replace and Labour Hours Determination

ABSTRACT

The present invention relates to the determination of repair operations for a damaged vehicle. More particularly, the present invention relates to determining repair operations, for example whether to repair or replace parts of a damaged vehicle and associated labour time required, for a damaged vehicle using images of the damage to the vehicle. 
     Aspects and/or embodiments seek to provide a computer-implemented method for determining repair operations that are required to repair a damaged vehicle, using images of the damage to the damaged vehicle.

FIELD

The present invention relates to the determination of repair operationsfor a damaged vehicle. More particularly, the present invention relatesto determining repair operations, for example whether to repair orreplace parts of a damaged vehicle and associated labour time required,for a damaged vehicle using images of the damage to the vehicle.

BACKGROUND

Typically, and as shown in FIG. 1, when a vehicle is involved in anaccident (or is damaged) 105, the vehicle or its driver will be insured,and the driver will contact the relevant insurance company 110 to make aclaim following a typical claim procedure 100.

The insurance company's estimation team 135 will then need to assess thedamage to the vehicle and approve any claim, and the driver or insurerwill then arrange for the vehicle to be repaired 145. Alternatively, theinsurance company may make a cash settlement 150 in place of arrangingor paying for repairs or may make a decision that the vehicle is a totalloss 140 and compensate the insured party accordingly or arrange for areplacement vehicle to be procured.

As shown in FIG. 1, the claim procedure 100 following an accident 105requires the driver or insured party to call their insurer 110, andpersonnel at the insurer will follow a script 115 to receive and processthe claim.

As part of the script 115, the insurer will obtain from the driver orinsured party some information about the accident 105. Typically, theinsurer will be provided with information about the insured person 120(which may also include details of the vehicle and its condition etcthat are provided during the call, or which are stored in the insurer'sdatabase and retrieved following receipt of the details of the insuredperson); details of the crash or accident 125, for example thecircumstances and extent of the damage; and photos of the damage 130.

The photos of the damage 130 are typically taken by the driver or theinsured party and can be of varying quality and comprehensiveness.Typically, photos are taken using phones equipped with cameras. Variousproblems can arise from this approach, including that too few photos aretaken and provided to the insurer. Also, the photos taken may not besufficiently well composed or may be of low quality due to the qualityof the camera used to take the photos or the skill of the user.

The photos of the damage 130 can be provided to the insurer either viae-mail, facsimile or post, for example. This means there is typically adelay in the receipt of the photos 130 by the insurer, thus delaying theprocessing of the claim by the insurer and slowing down thedecision-making process as to whether the damage is a total loss 140, orwhether a cash settlement 150 can be offered, or whether to arrange orallow the driver or insured part to arrange for repairs to the vehicle145.

As part of the claim procedure, and more specifically the claim reviewprocedure which is carried out by the insurer to verify the costs of theproposed repair work by manually assessing data provided by the clientand any proposed repairer, the insurer may request further informationor claim data to be provided from the driver or insured party regardingthe accident. This may include details of the vehicle and its conditionprior to any damage etc. These are typically provided during a telephonecall or are obtained having been stored in the insurer's database, butsometimes requires the insurer to contact the insured party in a followup telephone call, letter or e-mail requesting the further details.Further, the insurer will require sufficient details of the accident tobe provided, along with sufficient photographs of the damage forexample, so this must be obtained during the first and any subsequentcontact with the insured party. The process of obtaining sufficientinformation can be slow, especially if further requests for informationare made in separate subsequent contacts with the insured party, andthus can significantly delay the processing of an insurance claim.Further, the proposed repairer may be required to send details of theproposed repairs, including for example the labour tasks as well as anyparts or materials costs, to the insurer for approval prior tocommencing work. The insurer can then assess whether the claim iscovered by the relevant policy under which the claim is made anddetermine whether the estimated costs of repair can be verified and/orapproved as may be appropriate.

Various tools and processes have been developed to assist vehicle repairbusinesses and vehicle insurers respectively to prepare and approverepair proposals for damaged vehicles, for example as a result of thevehicle being involved in an accident.

Vehicle repair businesses need to be able to itemise both the labourrequired and the specific parts required in order to repair the vehicle,and then submit this for approval to an insurer where the repair iscovered by an insurance policy. Due to the large number of differentpossible makes and models that might require repair, and the optionalextras that might have been fitted to the vehicle to be repaired,vehicle repair businesses typically have to use a commercial database toidentify the correct make, model, year of manufacture and options fittedin order to correctly identify the parts that would need to be orderedif any need replacement.

Insurers typically require vehicle repair businesses to submit evidenceof the damage to each vehicle and a detailed repair proposal thatitemises the parts to be ordered and the respective costs of each partalong with detailed itemisation of the labour tasks and time that willbe required to carry out any repairs or replacement of parts. Preparingsuch detailed repair proposals manually typically takes vehicle repairbusinesses a significant amount of time.

In different jurisdictions, different approaches are taken by bothvehicle repair businesses (in respect of how repairs are carried out,what labour is deemed to be required, and preferences as to whether torepair or replace parts, for example) and insurers (in respect of whatpolicies are applied when approving or rejecting proposed repairs, forexample), depending on a variety of factors such as commercialpressures, regulation, consumer preference and typical insurancecoverage. Thus, detailed repair proposals will differ betweenjurisdictions and what insurers are prepared to approve in a detailedrepair proposal will also differ between jurisdictions.

Insurers, however, typically perform manual reviews on proposed repairsthat are submitted for approval by vehicle repair businesses. As aresult, the manual review process either requires a large workforce toperform the task of reviewing each submitted repair proposal or becomesa bottleneck in the repair approval process. For vehicle repairbusinesses, manual review can result in several disadvantages includingdelay in being able to begin repair work; further delays if the repairproposal is rejected by the insurer; and having to store customervehicles for longer periods than necessary resulting in both higherstorage space requirements and a higher probability of dissatisfiedcustomers.

Across all jurisdictions, a variety of the above-described problems canresult from manual preparation of proposed vehicle repairs and manualreview of the proposed vehicle repairs by insurers.

Improvements to the claim procedure would enable repairs to be completedsooner and for insurers to reach decisions faster and more efficiently.

SUMMARY OF INVENTION

Aspects and/or embodiments seek to provide a computer-implemented methodfor determining repair operations that are required to repair a damagedvehicle, using images of the damage to the damaged vehicle.

According to a first aspect, there is provided a computer-implementedmethod for determining repair operations to a damaged vehicle,comprising the steps of: receiving a plurality of images of the vehicle;determining, per part, one or more classifications for each of theplurality of images using one or more trained models, wherein eachclassification comprises at least one indication of damage to at leastone part; determine using the one or more classifications, using apre-determined threshold, one or more repair operations for each of theparts of the vehicle; and outputting the determined one or more repairoperations for each of the parts of the vehicle.

Determining repair operations using images of damage to a vehicle canallow for faster and more repeatable assessments of repair requirementsto damaged vehicles, for example to determine whether parts need to bereplaced or repairs, and for example to determine the amount of labourhours that will likely be required to make any repairs or replace anyparts.

Optionally, there is further performed the steps of: receiving vehicleinput data, wherein the vehicle input data comprises details of one ormore proposed parts and labour operations for repairing the damage tothe vehicle; and determining the parts and labour operations of thevehicle input data that are relevant to each of the plurality ofnormalised parts of the vehicle.

Receiving details of proposed repairs to a vehicle can allow for acomparison of these proposed repairs to the repair operations determinedfrom images of the damaged vehicle and a determination of which of theproposed repairs are deemed to be relevant to repairing the damage tothe vehicle (and, for example, which of the proposed repairs areirrelevant and/or superfluous and/or unnecessary).

Optionally, there is further performed the step of determining one ormore scores indicative of a damage value based each of the one or moreclassifications indicating damage to at least one part.

Determining scores for an indication of damage to each part and/or thevehicle can allow for easier processing and/or understanding of thedamage to the vehicle, and/or further processing of the damage statesand/or repair operations determined by the method.

Optionally, determining one or more repair operations for each of theparts of the vehicle comprises comparing the one or more scores to apre-determined threshold.

Determining whether the damage and/or repair operations determined fromthe images of the damaged vehicle meet, exceed or fall below apre-determined threshold can provide options for whether the damageand/or repair operations are necessary or should be performed (or iffurther operations should be performed that have been omitted from anyproposed repairs).

Optionally there is the use of any or any combination of: a computervision damage assessment model; a repair/replace prediction model; alabour hours prediction model; and/or a remove and install model.Optionally, the any or any combination of: a computer vision damageassessment model; a repair/replace prediction model; a labour hoursprediction model; and/or a remove and install model are used as one ormore secondary models that receive input from said one or more trainedmodels. Optionally, the one or more repair operations for each of theparts of the vehicle comprise replacing or repairing the damaged part ofthe vehicle. Optionally, there is further performed the step ofdetermining repair or replace labour hours based on the one or moreclassifications.

A variety of models/networks/archtectures can be used to determine anydamage and/or repair operations that might be required for a damagedvehicle from images of the damaged vehicle. Further, repair operationscan comprise determining repairs and/or part replacements and/or labourhours needed.

Optionally, the pre-determined threshold is adjustable based on one ormore jurisdictional requirements.

Allowing adjustment of the pre-determined threshold, manually orsubstantially automatically, can allow for the method to be adapted foruse for example in different jurisdictions/geographies and/or wheredifferent repair customs/rules apply.

Optionally, there is further performed the step of determining aplurality of parts of the damaged vehicle that are represented in theplurality of images of the damaged vehicle comprising the use of aplurality of classifiers. Optionally, each one of the plurality ofclassifiers is operable to detect each of the parts of the damagedvehicle. Optionally, there is further performed the step of determiningone or more sets of relevant images of each of the plurality of partsfrom the plurality of images of the vehicle. Optionally, there isfurther performed the step of determining a damage state of each of theparts of the vehicle using the one or more relevant images of each ofthe plurality of parts of the vehicle, the one or more damage statesbeing determined as one or more quantitative values.

Determining which parts are shown in the images of the damaged vehiclecan allow for the relevant images for each part to be identified and/orprocessed to determine damage state information and/or repairs thatmight be needed per part. A classifier can be trained per part tosubstantially optimise its performance for the part for which it wastrained. Outputting quantitative values can allow for downstream use ofthe output damage states for each part.

Optionally, the one or more trained models comprise one or more of: aneural network; a convolutional neural network; and/or a recurrentneural network.

A variety of model/network architectures and/or arrangements can be usedto implement the methods set out herein.

Optionally, there is further performed the step of querying one or moredatabases to determine any of: pre-painted vehicle part costs;additional cost for a pre-painted part; total paint time for apre-painted part; paint labour cost; material cost; and/or total paintcost.

Sending queries to databases, for example third party databases, torefine the output damage/repair classifications/determinations canprovide substantially more accurate outputs, for example to refine theoutput based on the specific make, model and year of the vehicleconcerned.

According to a further aspect, there is provided a computer-implementedmethod of training a neural network for determining repair operations toa damaged vehicle, including neurons, each neuron being associated withweights, wherein the method comprises: obtaining training inputs, thetraining inputs comprising images of the damaged vehicles and whetherany parts of the vehicles are damaged; for each training input,selecting one or more neurons based on their respective probability;adjusting the weights of the selected neurons such that the selectedneurons are substantially operable to classify per part of the vehicle,whether the part is damaged; processing the training input with theneural network to generate a predicted output; adjusting the weightsbased on the predicted output.

Training a model/network to determine the repair operations required,for example whether one or more parts need to be repairs and/or replacedand the associated labour hours per repair operation, can produce asubstantially accurate repair estimate for the repair operationsrequired from images of the damaged vehicle.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 shows a traditional insurance claim process according to theprior art;

FIG. 2 shows a near real-time insurance estimation process according toan embodiment;

FIG. 3 shows the process of determining an estimate for repair cost of adamaged vehicle from photos of the damaged vehicle and vehicleinformation according to an embodiment;

FIG. 4 shows a flowchart outlining a process of predicting whether thedamage to the vehicle is a total loss according to an embodiment;

FIG. 5 shows a flowchart outlining a process of estimating the repaircost of a damaged vehicle from photos of the damaged vehicle and vehicleinformation according to an embodiment;

FIG. 6 shows a flowchart illustrating in one embodiment that each parthas a dedicated classifier and that the photos for each part areprovided to a set of models for each part to produce initialestimates/pooled scores per part per model and then that the results areconcatenated into a damage signature vector according to an embodiment;

FIG. 7 shows a flowchart outlining a process of determining whetherpaint blending is required according to an embodiment;

FIG. 8 shows a flowchart outlining a process of determining whether apart is undamaged, in need of repair or in need of replacement accordingto an embodiment;

FIG. 9 shows a flowchart of a process for creating an estimate ofrepairs from photos input to the estimation method according to anembodiment;

FIG. 10 shows the structure of a neural network which can be used toprocess the input images and metadata according to an embodiment;

FIG. 11 shows a typical sequence of an insurance claim for vehicledamage, where the damage to the vehicle is repaired;

FIG. 12 shows the sequence of an insurance claim process includingautomated verification of proposed repair costs according to a describedembodiment;

FIG. 13 shows a more detailed overview of the automated verification ofproposed repair costs used in the embodiment shown in FIG. 12;

FIG. 14 shows an overview of an automated assessment platform accordingto an embodiment;

FIG. 15 shows an overview of a comparison analysis process that can beused in an embodiment;

FIG. 16 shows an illustration of a comparison analysis process that canbe used in an alternative embodiment;

FIG. 17 show how a pre-checked vehicle repair claim can be prepared andapproved according to an embodiment;

FIG. 18 shows an illustration of a generalised method of verifying aclaim input according to an example embodiment;

FIG. 19 shows an illustration of an embodiment of an approach tonormalise parts of vehicles to assist with the verification process ofthe embodiments described herein;

FIG. 20 shows a more detailed process of preparing an estimated repairschedule and/or repair cost according to another example embodiment;

FIG. 21 shows an alternative representation of normalised parts for avehicle according to an embodiment;

FIG. 22 shows a paint check process according to an embodiment;

FIG. 23 shows different example rules for a variety of jurisdictions asto what repairs would be permitted and categorize these alongside therespective decision made by the automated system;

FIG. 24 shows a model architecture using a gradient booster tree withmultiple decision trees according to an embodiment;

FIG. 25 shows an equation illustrating a method of calculating paintcost according to an embodiment;

FIG. 26 shows an overview of a damage determination process according toan embodiment;

FIG. 27 shows the use of cropping to preserve image resolution in adamage determination process from images according to an embodiment;

FIG. 28 shows the use of cropping and segmentation in a damagedetermination process according to an embodiment;

FIG. 29 shows an overview of a damage determination process using amulti-image learning model according to an embodiment;

FIG. 30 show an overview of a damage determination process using amulti-image graph model according to an embodiment;

FIG. 31 shows the process of using a language model to extractinformation relevant to a determination of visual damage from images ofthe damage according to an embodiment;

FIG. 32 shows the process of injecting structure accident data into avisual damage determination model according to an embodiment;

FIG. 33 shows a method of predicting damage to grouped auxiliary partsof a vehicle where the grouped auxiliary parts are associated with anormalised part/panel of the vehicle according to an embodiment;

FIG. 34 shows an example multi-task architecture according to anembodiment;

FIG. 35 shows a method of jointly training across variably sizeddatasets using domain confusion loss according to an embodiment;

FIG. 36 shows a method of jointly training across variably sizeddatasets using a subset of the larger dataset representative of the mostcomparable data according to an embodiment; and

FIG. 37 shows an example domain confusion loss architecture usable withembodiments.

SPECIFIC DESCRIPTION

Referring to FIG. 2, an example embodiment within a process for aninsurer to handle an insurance claim for an accident will now bedescribed.

In the following embodiments, and in the aspects and/or embodimentsdescribed above, the term “part” can be used interchangeably to refer toa specific part of a vehicle (i.e. a specific part of a specific make,model and year of vehicle) or a generic part of a vehicle as well as toa “normalised” or “standardised” part generic to most and/or allvehicles or to a region or zone of a generic vehicle or class ofvehicle. Further details are provided in relation to embodiments of thisconcept in FIGS. 19 and 21 and in the associated description for thesefigures set out below.

The insurance claim process is outlined in a flowchart 200 and beginswith an accident 105 (or damage being discovered) involving a vehiclethat is insured.

The party that is insured, or a party contacting an insurer on behalf ofan insured entity or in respect of an insured vehicle, will then contactthe insurer 110. The representative taking the telephone call on behalfof the insurer will run through a standard set of questions, or adecision tree of questions, to obtain a standard required set ofinformation about the accident 120, 125, 130 (e.g. the name of theinsured person/vehicle; details of the vehicle including make, model andcondition; details of any insurance policy; standard security questions;details of what happened to cause the accident; etc).

The phone call 110 and the party phoning the insurer providing theinformation requested by the insurer representative following the script115 will mean that details of the insured person 120 (or insuredparty/vehicle) will be provided to the insurer as well as crash details125 providing information about the damage/accident.

During the call, the representative will ask the person phoning theinsurer to provide photographs of the accident (and specifically photosof the damaged vehicle) 130 using the web app 205. If the person doesnot already have access to the web app 205, this can be provided by therepresentative by reading out a shortened URL or by sending a textmessage or email or other notification to a device of the person phoningthe insurer. Once the web app 205 has been loaded on the person'sdevice, it can be used to take photographs using the device imagingfunction (e.g. a camera in a smart phone) to provide photos 130 to theinsurer. The photos 130 will appear to the representative on theircomputer display shortly after these arrive at the insurer computersystems, and the representative can then guide the person on the ‘phoneto take further photos.

In alternative embodiments, other means of providing photos to theinsurer in parallel to the communication with the insurer can be used,so for example a chat interface can be used to communicate between theinsured party and the insurer and this can be adapted to allowphotographs to be taken and/or uploaded via the chat interface.

In this embodiment, the insurer provides a web application 205 that canbe used by the person phoning the insurer 110 by accessing the webapplication 205 using a URL in a standard browser and taking photos 130which are uploaded via the web application 205 and transmitted via theInternet to the insurer.

As the details of the insured person 120, crash details 125 and photosof the damage 130 are provided by the party phoning the insurerrepresentative, the computer system used by the representative isconfigured to display a dynamic live estimate of the repair cost of thedamage to the vehicle which is updated periodically as each item of newinformation is received and processed by the insurer computer system(either by the representative entering it into the computer system asthey follow the script for such calls or by receiving information fromthe insurer database about the relevant policy or by receiving photos ofthe damage via the web application 205).

The insurer representative can see a live damage estimate 210 displayedon the screen, which is determined using the method described in thefollowing embodiments based on the provided accident/damage photos 130,and in some embodiments may also be presented with a level of certaintyfor that estimate 210 and whether any further photos of any damage arerequired to increase the level of certainty in the estimate. In someembodiments, a required/threshold level of certainty is required inorder for the representative to authorise one or more of the possiblesolutions for the insured party (i.e. total loss 140, repair 145 or acash settlement 150).

Once there is an acceptable level of certainty in the estimate 210, orthe representative has concluded their script 115, the three optionstypically available to the representative are able to be chosen oroffered by the representative if permitted (i.e. total loss 140, repair145 or a cash settlement 150).

On the representative display, a total loss calculation 140 is made by atrained model which indicates whether, based on the information 120,125, 130 provided by the party calling the representative 110, itappears that the value of the likely repairs to the vehicle indicatethat the vehicle is a total loss. The output of this total losscalculation 140 is a yes/no indication to the representative. The outputcan be presented to the representative while they are still on thetelephone call with the person phoning the insurer, and the verificationof the calculation can be performed by the representative or otherinsurer personnel during the telephone call. In some embodiments, aninitial decision can be provided over the phone call but confirmationwill be made in some or all such decisions by insurance loss adjusters.

On the representative display, a cash settlement calculation 150 is madeby a trained computer model and displayed to the representative whichindicates whether, on the information 120, 125, 130 provided by theparty calling 110 the representative, an amount has been determinedaccording to the rules of the insurer that can be offered to the partycalling the representative as a cash settlement amount. The output ofthis cash settlement calculation 150 is a monetary amount that can beoffered by the representative for immediate payment in settlement of theclaim in lieu of arranging and paying for repairs to the vehicle. Theoutput can be presented to the representative while they are still onthe telephone call with the person phoning the insurer, and theverification of the calculation can be performed by the representativeor other insurer personnel during the telephone call, and the offer canbe made by the representative or other insurer personnel during thetelephone call and accepted by the caller during the telephone call. Inother embodiments, the offer can be presented as an initial decision andthe offer finalised by the insurer following the call and issued inwriting to the insured party to accept in writing.

On the representative display, the option to submit the informationprovided 120, 125, 130 (optionally along with the estimated and/or anauthorised damage repair cost) to a repair shop or body shop and toauthorise the repair of the vehicle 105 is also presented. In someembodiments, the estimated repair cost is provided to the repair shop orbody shop. In some embodiments, the estimated repair cost, or a figurebased on the estimated repair cost, is provided to the repair shop orbody shop as the authorised damage repair cost (i.e. repair costs inexcess of the authorised amount requires further approval from theinsurer before the repair shop or body shop can proceed with making anyrepairs).

The representative can offer a total loss option if permitted (i.e. ifthe system determines that the vehicle is a total loss), a cashsettlement option if permitted (i.e. if permitted by the insurer rules),or authorise the repair if permitted (i.e. if the vehicle is not a totalloss and the claim is valid and enough information has been provided).

The web app 205 in this embodiment can be provided as a computerapplication that is executed within a standard browser and accessibleusing a web URL, which allows anyone with a browser capable computingdevice (e.g. a smartphone, tablet computer, laptop or desktop computer)to access the web app while on a call with the insurer 110.Alternatively, it can be provided as or via a native application for asmartphone, tablet computer or a laptop/desktop computer.

Referring to FIG. 3, an example embodiment is shown of a method ofprocessing photos of a damaged vehicle in order to predict damage to avehicle and estimate the repair cost for the damaged vehicle 300 andthis method will now be described.

One or more photos of a damaged vehicle 130, typically in the region often to twenty photos, are provided by the caller to the insurer. Thesephotos 130 are provided to the damage estimation process shown in thisembodiment. The photos 130 will be of the damaged area of the vehicleprimarily, but perhaps from a few angles and from different distances(for example, to show a close-up photo of any damage as well as a morecontextual photo showing the damaged area and the surrounding undamagedportions of the vehicle).

In addition, vehicle information 131 is provided to the damageestimation process shown in this embodiment. Vehicle information caninclude model information, specifics of the vehicle (includingcondition, colour, optional features chosen when manufactured,modifications made to the vehicle compared to the standard). Imagescaptures of this vehicle in this embodiment (and other embodiments)would include images showing the overall vehicle (for example, imagestaken from at least roughly all four corners of the vehicle) as well asimages showing any damage to the vehicle.

The damage estimation process uses the photos 130 and vehicleinformation 131 in a computer vision analysis process 305, using atrained computer vision model. The computer vision analysis process 305identifies the parts of the vehicle shown in the photos 130, optionallyusing the vehicle information 131, using a set of multiple trainedmodels, each of which trained models having been trained to recognise aspecific and/or generalised vehicle part (e.g. a front bumper, or ahood, or left hand front door). The output of each of these models iscombined by the computer vision analysis process 305 into a damagevector. The damage vector combines the prediction output by each trainedmodel for each part, indicating a prediction of whether each part isdamaged, and/or an indication for each part of how damaged that part isand/or in what way it is damaged, and/or along with a confidence valuefor each part representing the certainty of the prediction made for eachpart. In this embodiment, the damage vector 310 comprises the combinedpredictions output by the trained models 305 along with a confidencevalue. In other embodiments the damage vector 310 can include additionalor alternative outputs from the computer vision process 305 and/or othertrained models processing the input photos 130 and/or vehicleinformation input 131.

The damage vector 310 is output by the computer vision process 305 into(a) a representative/caller confirmation process 315 and (b) a blendanalysis process 320 and (c) a repair/replace analysis process 325.

In some embodiments, where portions or all of the damage vector 310indicate a low confidence prediction, the representative/callerconfirmation process 315 displays to the representative if there are anylow confidence predictions (for example, having confidence values belowa predetermined threshold) for whether a part is damaged to allow therepresentative to guide the caller to take additional photos 130 of theone or more parts/regions of the damaged vehicle for which there are lowconfidence predictions. Alternatively, or in addition, therepresentative can be prompted during the representative/callerconfirmation process 315 to ask certain questions, depending on whichpart has a low confidence prediction, and the representative can inputthe answers to the representative/caller confirmation process 315 inorder to increase the prediction confidence above an acceptablethreshold by augmenting the input data, allowing a revised prediction tobe generated by the one or more trained models. The questions that areprovided to the representative by the representative/caller confirmationprocess 315 can be obtained from a database (not shown) of questionsthat can be asked per part/region/normalised part of the damaged vehicleand/or from additional photos of the damaged vehicle.

The blend analysis process 320 includes a set of multiple trainedmodels, each trained to perform blend analysis per part of the vehicleusing the damage vector 310 and the photos 130 and the vehicleinformation 131, along with any further information from therepresentative/caller confirmation process 315. Specifically, for eachpart of interest the blend analysis process 320 assesses whether anyneighbouring parts are damaged and will be repainted, or replaced with apre-painted part, in order to determine whether the part underconsideration will need to have a paint blending operation performed inorder to blend the paint on the part under consideration with the paintapplied to the repaired or replaced part(s) neighbouring the part underconsideration. In some embodiments, all parts will be considered,whereas in other embodiments only selected or determined parts will beconsidered by the blend analysis process.

The repair/replace analysis process 325 similarly uses a set of multipletrained models, each model trained to perform an assessment of aspecific part of a vehicle (be this a specific part, or region, ofnormalised part of the damaged vehicle). Specifically, each modelassesses, based on the input damage vector 310 and the photos 130 andthe vehicle information 131, along with any further information from therepresentative/caller confirmation process 315, whether the part underconsideration needs to be replaced or repaired.

To produce an output estimation of damage and the predicted repair costfor the vehicle, the outputs of the blend analysis process 320 and therepair/replace analysis process 325 are combined with data obtained by apart lookup process 326 by the output estimation process 330. The partlookup process 326 uses the input vehicle information 131 to determinethe current market prices for each part for the vehicle underconsideration and provides this to the output estimation process 330.The output estimation process 330 combines the predictions for damage toeach part of the vehicle, whether each damaged part needs to be repairedor replaced, the current prices for parts to be replaced, and anypainting/blending cost. The output of the output estimation process 330is a prediction for the repair cost for the damaged vehicle. Inalternative embodiments, the output is a prediction of the repairsnecessary, optionally broken down into materials and/or repairoperations required.

The photos 130 may be all of the photos provided to the insurer by acaller or may be a selection of those images either selectedautomatically (e.g. based on quality of image, image content, imagemetadata, a provided image description or caption from therepresentative or caller, or insurer rules) or manually (i.e. selectedfor use by the representative or other personnel of the insurer, orhighlighted by the representative or user as relevant).

The vehicle information 130 may sometimes be only a very limited amountof information such as the make and model of the vehicle, its age andgovernment registration details, and its paint colour, or it can be aricher dataset including comprehensive details about the vehiclespecification and condition as well as caller-provided andinsurer-collected information about the vehicle. Different embodimentshave different minimum input data requirements in respect of the vehicleinformation 130.

Different trained models and trained model arrangements can be used inthe computer vision analysis process 305 and/or the blend analysis 320and or the repair/replace analysis 325, and different arrangements forthese models can be implemented such as to train one or a plurality ofmodels that can recognise two or more parts rather than just onespecific part as per the above described embodiment.

During the representative/caller confirmation process 315, theinformation on low confidence predictions from the damage vector 310 canbe presented to either the representative or directly to the caller viathe web app in order to obtain either further photos from the caller orinformation to increase the confidence of the predictions for whichthere is low confidence output from the computer vision process 305.

In alternative embodiments, there is optionally an additional paintestimation process included as part of the method of processing photosof a damaged vehicle in order to predict damage to a vehicle andestimate the repair cost for the damaged vehicle. The inputs can includesome of the outputs from other processes. The paint estimation processcan use multiple trained models to assess, per part, whether anypainting is required for each part of the damaged vehicle or can obtainthis data from a third-party database. The output of these models isprovided to the output estimation process 330 which uses this additionalinformation to output the prediction for the repair cost for the damagedvehicle.

In alternative embodiments, there is optionally an additional labourestimation process as part of the method of processing photos of adamaged vehicle in order to predict damage to a vehicle and estimate therepair cost for the damaged vehicle. The labour estimation process canuse multiple trained models to determine, per part, the extend of thedamage and whether any labour is required to perform repairs and/orreplace any parts. A lookup can then be performed to produce an outputvalue by using the determination to obtain this output data from athird-party database, for example to determine an output value bysubmitting a query for a time value for one or more labour tasks torepair and/or replace partsalong with some or all of the vehicleinformation input 131 and receiving in return from the third-partydatabase repair information determined for that job for the specificmake, model and year (MMY) of the damaged vehicle. Some of the outputsfrom other estimation processes can be used as inputs to the labourestimation process. The output of these models is provided to the outputestimation process 330 which uses this additional information to outputthe prediction for the repair cost for the damaged vehicle.

In alternative embodiments, there is optionally an additional paintlabour estimation process as part of the method of processing photos ofa damaged vehicle in order to predict damage to a vehicle and estimatethe repair cost for the damaged vehicle. The paint labour estimationprocess can use multiple trained models to determine, per part, whetherno paint is needed, a “spot” paint job is needed, or more or less than50% of the part needs to be painted. A lookup can then be performed toproduce an output value by using the determination to obtain this outputdata from a third-party database, for example to determine an outputvalue by submitting a query for a “spot” paint job along with some orall of the vehicle information input 131 and receiving in return fromthe third-party database repair information determined for that job forthe specific make, model and year (MMY) of the damaged vehicle. Some ofthe outputs from other estimation processes can be used as inputs to thelabour estimation process. The output of these models is provided to theoutput estimation process 330 which uses this additional information tooutput the prediction for the repair cost for the damaged vehicle.

In some embodiments, there is optionally an additional remove andinstallation estimation process as part of the process of estimating arepair cost for the damaged vehicle. This estimation process estimatesthe amount of time that is necessary to perform all of the tasksrequired for replacing a damaged part of a vehicle. These estimates areobtained from a rule set for each part or group of parts/type of partthat has been built up by an expert based on knowledge of repairs tovehicles, or are obtained from a third party data provider to which aquery specifying the part in question and the vehicle information andsome damage information is transmitted and an estimate returned by thedata provider. Optionally, this process receives one or more outputsfrom other modules that are included in its output. The output from therule set or the returned estimate from the data provider is then used asthe output from the remove and installation process.

In other embodiments, there is optionally an additional overlapcalculation performed. The overlap that can be identified is the overlapin repair work to be performed when considering repairs to neighbouringparts that are damaged, as repairs to multiple neighbouring parts willinvolve common components that only need to be replaced/handled or tasksthat can be performed once and thus do not need to be repeated for eachpart. In some embodiment, this overlap is determined from a query to athird-party data provider. Thus, a modifier can be determined based onthe number and degree of overlap between repair tasks and components totake into account the overlap between repairs to neighbouring damagedparts of a vehicle.

Referring to FIG. 4, an example embodiment 400 of a method of predictingtotal loss based on the input information and photos provided by acaller to the insurer and the representative will now be described.

In this embodiment, a method 400 of predicting total loss is describedthat can be used with any of the other described embodiments oralternative embodiments, or with the aforementioned aspects described inthis specification.

According to this embodiment, the method 400 takes as input data thecrash and insurance details 405 and the repair estimate 410 output bythe damage prediction model of one of the embodiments described hereinand the vehicle value 415.

The crash and insurance details 405 will be provided by the caller tothe insurer's representative when calling or contacting the insurer toreport the accident/damage to the damaged vehicle as described inrelation to the other described embodiments herein.

The estimate for repair costs for the damaged vehicle 410 is determinedusing the damage repair estimation process described in the embodimentsand alternative embodiments that have been described herein. Thisestimate 410 may comprise any combination of: the operations required torepair the damaged vehicle; the parts and/or materials required torepair the damage vehicle; and the associated financial costs ofrepairing the damaged vehicle.

The vehicle value 415 may be estimated, and/or have been provided by theinsured party, and/or be determined by the insurer using rules or otherguidelines, and/or is provided by a third party.

The prediction of total loss 420 is then determined, using a trainedmodel, from a variety of factors including whether the crash details 405indicate a class of accident that typically results in significantstructural damage to a car making it unsafe. In other embodiment, otherfactors/combinations of factors can be used to make this determination.If it is determined that there has been significant structural damage,for example, the assumption can be made that the accident is a totalloss. In alternative embodiments, the details of the accident can beassessed using a database of classes of accident (the database can begenerated using knowledge from experts, or using data from insurers, oraccessed through the use of a third part data provider) and/or thedamage assessment that forms part of the repair estimate 410 todetermine whether there is a high likelihood of a total loss or whetherthere is likely to be significant non-visible damage to the vehicle. Inother embodiments, the repair estimate is generated (including anestimate of the financial costs of the repair) as per the describedembodiments and if the total value of the repair is above a certainvalue, for example the replacement value or insured value of thevehicle, or a substantial fraction thereof, then it is determined thatthe accident has resulted in a total loss. The determination of totalloss 420 also considers the damage repair estimate 410 against thevehicle value 415 using the insurer's own rules, policies and/orprocedures to determine whether the cost of the repairs 410 is likely toexceed the vehicle value 415 or a significant proportion of the vehiclevalue 415.

Referring to FIG. 5, another example embodiment is shown of a method 500of processing photos of a damaged vehicle in order to predict damage toa vehicle and estimate the repair cost for the damaged vehicle, and thismethod 500 will now be described.

One or more photos of a damaged vehicle 130, typically in the region often to twenty photos, are provided by the caller to the insurerrepresentative. These are provided to the damage estimation process 500shown in this embodiment. The photos 130 will be of the damaged area ofthe vehicle primarily, but typically from a few different viewing anglesand from different distances (for example, to show a close-up photo ofany damage as well as a more contextual photo showing the damaged areaand the surrounding undamaged portions of the vehicle).

In addition, vehicle information 131 is provided to the damageestimation process shown in this embodiment. Vehicle information 131 caninclude model information, specifics of the vehicle (includingcondition, colour, optional features chosen when manufactured,modifications made to the vehicle compared to the standard).

The damage estimation process uses the photos 130 and the vehicleinformation 131 in a computer vision analysis process 305. The computervision 305 identifies the parts of the vehicle shown in the photos 130,optionally using the vehicle information 131, and uses of a set ofmultiple trained models, each of which trained models has been trainedto recognise a specific generic vehicle part or region (e.g. a frontbumper, or a hood, or left hand front door). The output of each of theseper part/region models is combined by the computer vision analysisprocess 305 into a damage vector 310. The damage vector 310 combines theprediction output by each trained model for each part/region indicatinga prediction of whether each part/region is damaged, and an indicationfor each part/region how damaged that part/region is, along with aconfidence value for each part/region representing the certainty of theprediction made for each part/region. In other embodiments, specificparts for a specific make, model and year (MMY) of a vehicle arerecognised instead of regions/generalised parts/normalised parts/genericregions, optionally by using both the photos 130 and the vehicleinformation 131 rather than just the photos 130.

The damage vector 310 is output by the computer vision process 305 into(a) a representative/caller confirmation process 315 and to (b) a blendanalysis process 320 and (c) a repair/replace analysis process 325 and(d) a paint analysis process 505. Alternatively, the paint analysisprocess 505 can have its input as one or more of the outputs from one ormore of the other processes 320, 325.

The representative/caller confirmation process 315 displays to therepresentative if there are any low confidence predictions (i.e. under apredefined threshold confidence value) for whether a part/region isdamaged, to allow the representative to guide the caller to takeadditional photos 130 of the one or more parts/regions for which thereare low confidence predictions. Alternatively, or in addition, therepresentative can be prompted during the representative/callerconfirmation process 315 to ask certain questions, depending on whichpart/region has a low confidence prediction, and the representative caninput the answers to the representative/caller confirmation process 315,which information is provided to the model 305 in order for the model torecalculate any low-confidence predictions, in order to increase theprediction confidence above an acceptable threshold. The questions thatare provided to the representative by the representative/callerconfirmation process 315 can be obtained from a database (not shown) ofquestions that can be asked per part/region of interest.

The blend analysis process 320 includes a set of multiple trainedmodels, each trained to perform blend analysis per part/region of thevehicle using the damage vector 310 and the photos 130, along with anyfurther information obtained from the representative/caller confirmationprocess 315. Specifically, the blend analysis process 320 assesseswhether any neighbouring parts/regions are damaged, and thus will berepainted or replaced with a pre-painted part, in order to determinewhether the part/region under consideration will need to have a paintblending operation performed in order to blend the paint on thepart/region under consideration with the paint applied to the repairedor replaced part/regions(s) neighbouring the part/region underconsideration.

The paint estimation process 505 uses multiple trained models to assess,per part/region, whether any painting is required for each part/regionof the damaged vehicle. The input to the paint estimation process 505 isthe damage vector 310 (and, in some embodiments, the photos 130), alongwith any further information obtained from the representative/callerconfirmation process 315. The output of these models is provided to theoutput estimation process 330.

The repair/replace analysis process 325 again includes a set of multipletrained models, each trained to perform an assessment of a specificpart/region of a vehicle. Specifically, each model assesses, based onthe input damage vector 310 and the photos 130, along with any furtherinformation obtained from the representative/caller confirmation process315, whether the part/region under consideration needs to be replaced orrepaired.

To produce an output estimation of damage and the predicted repair costfor the vehicle, the outputs of the blend analysis process 320 and therepair/replace analysis process 325 and the paint analysis process 505are combined with data obtained by a part lookup process 326 by theoutput estimation process 330. The part lookup process 326 uses theinput vehicle information 131 to determine the current market prices foreach part, or associated parts for the region, for the vehicle underconsideration and provides this to the output estimation process 330.The output estimation process 330 combines the predictions for damage toeach part/region of the vehicle, whether each damaged part/region needsto be repaired or replaced, the current prices for parts to be replaced,and any painting/blending cost. The output of the output estimationprocess 330 is the prediction for the repair cost for the damagedvehicle.

The photos 130 may be all of the photos provided to the insurer by acaller or may be a selection of some of those images, either selectedautomatically (e.g. based on quality of image, image content, imagemetadata, a provided image description or caption from therepresentative or caller, or insurer rules) or manually (i.e. selectedfor use by the representative or other personnel of the insurer, orhighlighted by the representative or user as relevant).

The vehicle information 130 can be only a very limited amount ofinformation such as the make and model of the vehicle, its age andgovernment registration details, and its paint colour, or it can be aricher dataset including comprehensive details about the vehiclespecification and condition as well as caller-provided andinsurer-collected information about the vehicle.

Different trained models and trained model arrangements can be used inthe computer vision analysis process 305, and different arrangements forthese models can be implemented such as to train one or a plurality ofmodels that can recognise a generalised part/region, or two or moreparts, rather than just one specific part in the above describedembodiments.

During the representative/caller confirmation process, the informationon low confidence predictions from the damage vector 310 can bepresented to either the representative or directly to the caller via theweb app in order to obtain either further photos from the caller orinformation to increase the confidence of the predictions for whichthere is low confidence output from the computer vision process 305.

Alternatively, a further labour estimation process (not shown) canoptionally also be included in embodiments and used as part of theoutput estimation process 330 to determine a prediction for the repaircost for the damaged vehicle.

In some embodiments, there is also a remove and installation estimationprocess (not shown) optionally included as part of the process ofestimating a repair cost for the damaged vehicle. This estimationprocess estimates the amount of time that is necessary to perform all ofthe tasks required for replacing a damaged part of a vehicle. Theseestimates are obtained from a rule set for each part or group ofparts/type of part that has been built up by an expert based onknowledge of repairs to vehicles, or are obtained from a third partydata provider to which a query specifying the part in question and thevehicle information and some damage information is transmitted and anestimate returned by the data provider. The output from the rule set orthe returned estimate from the data provider is then used as the outputfrom the remove and installation process.

In other embodiments, there is optionally also an overlap calculationperformed (not shown). The overlap that can be identified is the overlapin repair work to be performed when considering repairs to neighbouringparts that are damaged, as repairs to multiple neighbouring parts willinvolve common components that only need to be replaced/handled or tasksthat can be performed once and do not need to be repeated for each part.Thus, a modifier can be determined based on the number of, and degree ofoverlap between, repair tasks and components to take into account theoverlap between repairs to neighbouring damaged parts/regions of avehicle.

Referring to FIG. 6, an example embodiment of a process 600 of takingthe photos 130 provided by the caller and processing these to output adamage signature vector 535 will now be described.

As in the other described embodiments, the caller provides photos 130 ofthe damaged vehicle. Typically these might number in the region of 10 to20 images, but fewer photographs may be supplied (for example where onlyminor damage is visible) or conversely more photographs may be supplied(for example where there is extensive damage that requires manyphotographs to document and/or the caller provides a comprehensiveselection of close up and distant/further removed photos of the damagedvehicle).

The part classifier 505 is a set of multiple models 505 i, 505 ii, 505iii, 505 iv to 505 n (n being the number of models) each of which runsin parallel on each of the photos 130 and each of which models 505 i,505 ii, 505 iii, 505 iv to 505 n is trained to determine whether aphotograph contains a specific part/region of a vehicle. The output ofeach of the models 505 i, 505 ii, 505 iii, 505 iv to 505 n are thedeterminations whether each part/region (for which each model is trainedto detect) is present in each of the photos and some metadata, forexample optionally indicating the damage predicted for that part and/oran associated confidence value.

The output from each of the multiple models 505 i, 505 ii, 505 iii, 505iv to 505 n of the part classifier 505 is fed into each of a set offurther assessment models per part along with the photos 130. Toillustrate the next stages in the process/method, only the output fromone model 505 i will be described here but the reader will understandthat each of the multiple models 505 i, 505 ii, 505 iii, 505 iv to 505 nof the part classifier 505 creates an output that is fed to a like groupof assessment models per part in parallel with the described process inrelation to the output from one model 505 i that will now be described.

The output from the first models 505 i of the part classifier 505 andthe photos 130 is fed into multiple further assessment models,specifically a paint assessment model 511, a blend assessment model 510,a repair or replace determination model 515 and a labour estimationmodel 520.

The paint assessment model 511 receives the photos 130 and determineswhich of the photos 130 are deemed relevant by the part classifier model505 i (i.e. show the part of interest) along with the output metadataindicating the damage predicted for that part and an associatedconfidence value. The paint assessment model 511, for each relevantphoto of the input photos 130, then predicts whether and the extent towhich painting is required for each photo of the damaged part. Thisprediction is then output for each relevant photo of the damaged partand an initial estimate is determined 513 based on a pooled score of allof the predictions output by the paint assessment model 511 for all ofthe photos of the damaged part provided as an input to the paintassessment model 511 by the part classifier model 505 i. Alternatively,the paint assessment model 511 can receive the outputs from other modelsand the paint assessment process submits queries to one or more thirdparty databases to obtain information on what paint costs and tasks arerequired.

The blend assessment model 510 receives the photos 130 and determineswhich of the photos 130 are deemed relevant by the part classifier model505 i along with the output metadata indicating any damage predicted forthat part and neighbouring parts and an associated confidence value. Theblend assessment model 510, for each relevant photo of the input photos130, then predicts whether and the extent to which blending is requiredfor each photo of the part being considered based on the damagepredicted for that or any neighbouring parts. This prediction is thenoutput for each relevant photo of the part being considered and aninitial estimate is determined 512 based on a pooled score of all of thepredictions output by the blend assessment model 510 for all of photosof the part being considered provided as an input to the blendassessment model 510 by the part classifier model 505 i.

The repair or replace determination model 515 receives the photos 130and determines which of the photos 130 are deemed relevant by the partclassifier model 505 i along with the output metadata indicating thedamage predicted for that part and an associated confidence value. Therepair or replace determination model 515, for each relevant photo ofthe input photos 130, then predicts whether repair is possible orwhether replacement is necessary for each photo of the damaged partbased on the damage predicted by the part classifier model 505 i forthat photo. This prediction is then output for each relevant photo ofthe damaged part and an initial estimate is determined 517 based on apooled score of all of the predictions output by the repair or replacedetermination model 515 for all of the photos of the damaged partprovided as an input to the repair or replace determination model 515 bythe part classifier model 505 i.

The labour estimation model 520 receives the photos 130 and determineswhich of the photos 130 are deemed relevant by the part classifier model505 i along with the output metadata indicating the damage predicted forthat part and an associated confidence value. The labour estimationmodel 520, for each relevant photo of the input photos 130, thenpredicts what labour for repair or replacement or painting of the partunder consideration for each photo of the part under consideration basedon the damage predicted by the part classifier model 505 i for thatphoto and any neighbouring parts. This prediction is then output foreach relevant photo of the part under consideration and an initialestimate is determined 522 based on a pooled score of all of thepredictions output by the labour estimation model 520 for all of thephotos of the part under consideration provided as an input to labourestimation model 520 by the part classifier model 505 i. Alternatively,based on the output of one or more of the other processes, the labourestimation model 520 submits queries to one or more third partydatabases to obtain labour task and price estimates.

The output initial estimates 513, 512, 517, 522 are then concatenated530, along with any other initial estimates produced for painting,blending, repair, replacement or labour for each of the other partsprocessed by the multiple models 505 i, 505 ii, 505 iii, 505 iv to 505 nof the part classifier 505. The output of the concatenation 530 is avector with classification 535 (otherwise known as a damage signature).

Referring to FIG. 7, an example embodiment of a blending assessmentmodel 700 will now be described.

The inputs to the blending assessment model 715 are the damage signatureoutput from the computer vision or part detection model 535 along withthe car details 705 and the information about the damage to the vehicle710 provided by the caller to the insurer representative.

The blending assessment model 715 works on a per part basis, so istrained to assess whether blending is required for a specificpart/region of a vehicle and uses information in the damage signature535 on what damage there is to neighbouring parts to assess whether anundamaged part/region requires blending with the new paint applied toeither a repaired part or a replaced part that is painted or comespre-painted. The model 715 is a trained machine learned model in thisembodiment.

The output 720 of the blending assessment model 715 is a binary outputper part as to whether blending is required for each part that isconsidered by each of the multiple per part blending assessment models715. In other embodiments, more detailed outputs can be produced.

Referring to FIG. 8, an example embodiment 800 of a repair or replacemodel (per part) 715 will now be described.

The repair or replace model (per part) 715 works on a per part basis, sois trained to assess whether a repair to each part is possible orwhether a replacement part is required for a specific part of a vehicle.The repair or replace model (per part) 715 uses information in thedamage signature 535 on what damage there is to the part beingconsidered to assess whether a part is undamaged or whether a part canbe repaired or requires replacing with a new part. The model 715 in thisembodiment is a trained machine learned model.

The output 720 of the repair or replace model (per part) 715 is anoutput per part indicating whether each part is considered by each ofthe multiple per part estimation models 715 to be undamaged, repairableor need replacing.

Referring to FIG. 9, an example embodiment 900 of a variant of themethod of producing a repair estimate from photos 130 of a damagedvehicle will now be described.

One or more photos of a damaged vehicle 130, typically in the region often to twenty photos, are provided by the caller to the insurerrepresentative. These are provided to the damage estimation processshown in this embodiment. The photos 130 will be of the damaged area ofthe vehicle primarily, but perhaps from a few viewing angles and fromdifferent distances (for example, to show a close-up photo of any damageas well as a more contextual photo showing the damaged area and thesurrounding undamaged portions of the vehicle).

The damage estimation process first uses the photos 130 in a computervision analysis process 305. The computer vision analysis process 305identifies the parts/regions of the vehicle shown in the photos 130using of a set of multiple trained models, each of which trained modelshas been trained to recognise a specific vehicle part/region (e.g. afront bumper, or a hood, or left hand front door). The output of each ofthese models is combined by the computer vision analysis process 305into a damage vector 310. The damage vector 310 combines the predictionoutput by each trained model for each part/region indicating aprediction of whether each part/region is damaged, and an indication foreach part/region how damaged that part/region is, along with aconfidence value for each part/region representing the certainty of theprediction made for each part/region.

The damage vector 310 is output by the computer vision process 305 into(a) a representative/caller confirmation process 315 and to (b) a blendengine 320 and (c) a repair/replace analysis process 325 and (d) aremove and install (R&I) process 910. In some embodiments, the damagevector 310 is also provided to a paint analysis process 505 and to alabour analysis process. In this embodiment, the outputs of the otherprocesses 320, 910, 325 are provided as inputs to the repair labouranalysis process 905 and paint analysis process 505.

The representative/caller confirmation process 315 displays to therepresentative if there are any low confidence predictions within thedamage vector 310, as to whether a part/region is predicted to bedamaged or not, to allow the representative to guide the caller to takeadditional photos 130 of the one or more parts for which there are lowconfidence predictions. Alternatively, or in addition, therepresentative can be prompted by the representative/caller confirmationprocess 315 to ask certain questions, depending on which part has a lowconfidence prediction, and the representative can input the answers tothe representative/caller confirmation process 315 in order to increasethe prediction confidence above an acceptable threshold. The questionsthat are provided to the representative by the representative/callerconfirmation process 315 can be obtained from a database (not shown) ofquestions that can be asked per part. The output from this process isprovided to the representative/caller confirmation process 315 and therevised output of the computer vision process 305, once the newinformation obtained during the confirmation process 315 has also beenobtained, is a revised damage vector 310 which is provided to the blendengine 320, the repair/replace analysis process 325, the paint analysisprocess 505, and to the labour analysis process (via the blend engine320). Alternatively, the output from this process is provided along withthe original damage vector 310 to the representative/caller confirmationprocess 315, the blend engine 320, the repair/replace analysis process325, the paint analysis process 505, and to the labour analysis process(via the blend engine 320).

The blend engine 320 (or blend analysis process) includes a set ofmultiple trained models, each trained to perform blend analysis perpart/region of the vehicle using the damage vector 310 and the photos130, along with any further information from the representative/callerconfirmation process 315. Specifically, the blend analysis process 320assesses whether any neighbouring parts/regions are damaged and will berepainted or replaced with a pre-painted part in order to determinewhether the part/region under consideration will need to have a paintblending operation performed in order to blend the paint on the partunder consideration with the paint applied to the repaired or replacedpart/region(s) neighbouring the part/region under consideration.

The paint estimation process 505 uses multiple trained models to assess,per part/region, whether any painting is required for each part/regionof the damaged vehicle. In some embodiments, the input to the paintestimation process 505 can be the damage vector 310 and the photos 130,along with any further information from the representative/callerconfirmation process 315 and/or a revised damage vector 310. In thisembodiment, the input is the outputs per part from the blend engine 320,the R&I analysis 910 and the repair/replace analysis 325. The output ofthese models is provided to the output estimation process 330. The paintestimation process 505 determines whether, for each part, no paint isneeded, a “spot” paint job is needed, or more or less than 50% of thepart needs to be painted. To obtain an estimate of the hours of labourneeded to perform the determined level or painting, and to obtain aprice for the paint required, one or more queries are sent to one ormore third party databases to obtain data or an estimate.

The repair/replace analysis process 325 again includes a set of multipletrained models, each trained to perform an assessment of a specificpart/region of a vehicle. Specifically, each model assesses, based onthe input damage vector 310 and the photos 130, along with any furtherinformation from the representative/caller confirmation process 315,whether the part/region under consideration needs to be replaced orrepaired.

The repair labour analysis module 905 takes input from the blend engine320 and the repair/replace assessment module 325 and the paint analysismodule 505 and makes a determination of the required labour time/amountneeded to make the determined/estimated repairs/replacements and anypaint application, and this determination is output to the estimationoutput process 305 for incorporation into the predicted repair cost.

In some embodiments, there is also an optional remove and installationestimation process 910 as part of the process of estimating a repaircost for the damaged vehicle. This estimation process 910 estimates theamount of time that is necessary to perform all of the tasks requiredfor replacing a damaged part/region of a vehicle. These estimates areobtained from a rule set for each part or group of parts/type of partthat has been built up by an expert based on knowledge of repairs tovehicles, or are obtained from a third party data provider to which aquery specifying the part in question and the vehicle information andsome damage information is transmitted and an estimate returned by thedata provider. The output from the rule set or the returned estimatefrom the data provider is then used as the output from the remove andinstallation process.

In other embodiments, there is also an optional overlap calculationperformed. The overlap that can be identified is the overlap in repairwork to be performed when considering repairs to neighbouring parts thatare damaged, as repairs to multiple neighbouring parts will involvecommon components that only need to be replaced/handled or tasks thatcan be performed once and do not need to be repeated for each part.Thus, a modifier can be determined based on the number and degree ofoverlap between repair tasks and components to take into account theoverlap between repairs to neighbouring damaged parts of a vehicle. Inthis embodiment, calculation of any modifications to values due to anyoverlap is performed by the repair labour analysis module 905.

To produce an output estimation of damage and the predicted repair costfor the vehicle, the outputs of the blend analysis process 320 and therepair/replace analysis process 325 and the paint analysis process 505and the repair labour analysis module 905 are combined by the outputestimation process 330 and any modified calculation is applied. Theoutput estimation process 330 combines the predictions for damage toeach part of the vehicle, whether each damaged part needs to be repairedor replaced, any painting/blending cost and the labour cost (all of someof which having had a modifier, if calculated, applied to reflect areduction due to overlap). The output of the output estimation process330 is the prediction for the repair cost for the damaged vehicle.

In some embodiments, a overlap engine is used to perform an overlapcalculation, to determine one or more reductions to be applied, forexample where two neighbouring parts require repair operations to beperformed, a reduction in labour time due to efficiencies fromperforming the repair operations/labour at the same time for bothneighbouring parts can be applied to the overall total repaircost/operations/materials.

The photos 130 may be all of the photos provided to the insurer by acaller or may be a selection of those images either selectedautomatically (e.g. based on quality of image, image content, imagemetadata, a provided image description or caption from therepresentative or caller, or insurer rules) or manually (i.e. selectedfor use by the representative or other personnel of the insurer, orhighlighted by the representative or user as relevant).

Different trained models and trained model arrangements can be used inthe computer vision analysis process 305, and different arrangements forthese models can be implemented such as to train one or a plurality ofmodels that can recognise two or more parts rather than just onespecific part as per the above described embodiment.

During the representative/caller confirmation process, the informationon low confidence predictions from the damage vector 310 can bepresented to either the representative or directly to the caller via theweb app in order to obtain either further photos from the caller orinformation to increase the confidence of the predictions for whichthere is low confidence output from the computer vision process 305.

In some embodiments, vehicle information 131 can also be provided to thedamage estimation process shown in this embodiment. Vehicle informationcan include model information, specifics of the vehicle (includingcondition, colour, optional features chosen when manufactured,modifications made to the vehicle compared to the standard). The vehicleinformation 130 can be only a very limited amount of information such asthe make and model of the vehicle, its age and government registrationdetails, and its paint colour, or it can be a richer dataset includingcomprehensive details about the vehicle specification and condition aswell as caller-provided and insurer-collected information about thevehicle. The vehicle information can be used by any of the computervision module 305, the blend engine 320, the paint estimation module505, the repair/replace analysis module 325 and the repair labouranalysis module 905 as well as for producing the final estimation output330. A part lookup process 326 can use the input vehicle information 131to determine the current market prices for each part for the vehicleunder consideration and provide this to the output estimation process330.

Referring to FIG. 10, an example embodiment 1000 of a trained modelarchitecture 1030 will now be described.

The trained model architecture 1030 is a neural network with a series oflayers 1060, 1070, 1080, 1090, 1100 between most of which layers thereare injection points 1075, 1085, 1095, 1105.

The input to the neural network 1030 is an image of a part 1010. Theneural network 1030 is also provided with metadata 1020, which is splitinto two groups of metadata 1040, 1050, and which metadata 1040, 1050 isinjected into the neural network 1030 between layers 1070, 1080, 1090,1100 at injection points 1075, 1085, 1095, 1105.

The metadata 1020, 1040, 1050 can include (i) the repair/replace scorespredicted for the part/region in question; (ii) the repair/replacescores predicted for any, multiple or all nearby parts/regions (iii) theundamaged/damaged values of the part/region in question (iv) theundamaged/damaged values of any, multiple or all nearby parts/regions;(v) the type of vehicle (e.g. whether the vehicle is a pick-up truck,car, van, etc); (vi) the number of doors the vehicle has (e.g. whetherthe vehicle has two, three or four doors); and (vii) the colour of thevehicle (e.g. whether the vehicle is white, or where there are specificcolours to types of vehicle, more precise colours or colour metadata canbe captured). Other metadata can be injected in other embodiments, forexample details about the vehicle (make, model and year; optionsselected, etc), damage (for example, the output of anyclassifiers/models), and accident circumstances (for example, priordamage, circumstances of the cause of the damage, etc).

In an embodiment where the neural network 1030 is being used as ablending assessment model per part/region, only items (ii), (iv), (v)(vi) and (vii) of metadata 1020, 1040, 1050 will be used.

Each of items (ii), (iv), (v) (vi) and (vii) of metadata 1020, 1040,1050 are injected separately at each of the injection points 1075, 1085,1095, 1105.

The output 1200 of the neural network 1030 depends on how it has beenconfigured. In an embodiment where the neural network 1030 is configuredas a blending assessment model, the output will be a binary yes or noanswer as to whether the part in question requires a paint blendingoperation to be performed thereupon.

Alternatively, instead of the neural network being constructed from aseries of single layers 1060, 1070, 1080, 1090, 1100, the neural networkcan be constructed from one or more groups of layers.

In other embodiments the neural network 1030 can be configured to be ablend engine, a repair/replace analysis process, a remove and installprocess, a modifier calculation process, a paint analysis process, or alabour analysis process. Different metadata 1020 will be used in each ofthese embodiments.

As shown in FIG. 11, which will be described in more detail below,conventionally vehicle repair cost estimates (or “claim estimates”) aregenerated at a body shop (i.e. a garage or workshop where vehicles areserviced and/or repaired and which has been proposed by either acustomer or insurer as the repairer of the insured damaged vehicle) whena customer brings in a damaged vehicle that is insured. The estimatesare then reviewed by the insurer (this is typically known as the “claimreview process”). Insurers, specifically a specialist team of people orrepresentatives working for the insurer, when reviewing claim estimates,may be tasked to follow pre-defined procedures or workflows. Suchprocedures or workflows may require the relevant people/representativesto routinely challenge claim estimates where statistical anomalies areidentified, for example. However, insurers face issues in reviewingclaims accurately and efficiently as the task when performed manually istime-intensive and prone to human error. Inefficient processes inreviewing claims can result in a backlog of claims to be reviewedmanually and can accumulate high costs associated with the inefficientprocesses. Providing an automated or semi-automated claim review systemcan help triage claims, and can help to identify leakage (i.e. proposedrepairs that are either not covered by the relevant insurance policy orare unnecessary, incorrectly priced or even fraudulent) in claims, usingautomated assessments of the data and information provided to theinsurer by both the insured party and the proposed repairer.

With reference to FIGS. 12 to 17, example embodiments relating toassessing claim input data using an automated assessment platform forverifying the claim input data, by generating estimates of damages anddetermining whether claim input data can be approved or is anomalous orwhether further assessment may be required, will now be described.

Example embodiments provide an automated system to assess or reviewwhether vehicle repair estimates are acceptable and legitimate. Thesystem can be used to flag repair operations that are likely to beirregular in some way (i.e. potential leakage), which can then befurther reviewed by the insurer (usually manually, by a specialistclaims reviewer or loss adjuster).

In some embodiments, where no irregular operations are identified, orthe number (or value) of irregular operations are below a pre-determinedthreshold, the repair operations can be automatically approved on behalfof the insurer, or allocated to a queue or workflow of the insurer.

As shown in FIG. 11, which shows a traditional vehicle repair claimprocess 1100, in cases of vehicle damage or following accidents 1105,the insured party will typically either take the insured vehicle to arepair shop or body shop for repair 1115 and will contact their insurerto report the accident 1110.

The repair shop or body shop may be chosen by the insured party or bythe insurer, depending on various factors such as but not limited to theterms of the relevant insurance policy and the preference of the insuredparty.

If the insured party takes the damaged vehicle to the repair shop 1115first, the damage is assessed by the repair shop in order for them toprepare a quote for, or an estimated cost of, the repair 1120 that isrequired to the damaged vehicle. Depending on the various factors, suchas the procedures of the repair shop, the policy of the insurer and theseverity of the damage or accident, the estimate from the repair shop issent to the insurer 1125 for authorisation by the insurer as typicallyinsurers scrutinise any repair estimates received 1130. The repair shopmay transmit their estimated repair cost along with supporting details(e.g. repair labour proposed, parts required, materials required such aspaint) to the insurer 1125. In addition to the repair estimate 1120being sent to the insurer by the repair shop, the party that is insured(or a party reporting the damage/accident on behalf of an insured entityor in respect of an insured vehicle) will contact their insurer 1110 toreport the accident or damage that has occurred and provide details tothe insurer about the accident or damage 1125 including providing imagesof the damaged vehicle (which are provided either by the driver or therepair shop).

The insurer will then review the claim, optionally requesting furtherdetails from the insured party and/or the repair shop as necessary toverify the claimed amounts before approving the repair work or sendingpayment to either the insured party or the repair shop.

Specifically, the representative taking the telephone call on behalf ofthe insurer, when the client reports the accident 1105 to the insurer1110, may run through a standard set of questions, or a decision tree ofquestions, to obtain a standard required set of information about theaccident (e.g. the name of the insured person/vehicle; details of thevehicle including make, model and condition; details of any insurancepolicy; standard security questions; details of what happened to causethe accident; etc). In this way the insurer can be provided withinformation, for example a standard checklist of information requestedby the insurer representative by following script, that will ensure thatcomprehensive details of the insured party and vehicle will be providedto the insurer as well as comprehensive accident and damage details.Additionally, photographs of the damaged vehicle may also be provided tothe insurer.

As the details of the insured person, accident details and photographicimages of any damage are provided by the party to the insurer, theinsurer can consider 1130 and scrutinise the details provided typicallyto assess, but not limited to, the following (a) whether there is avalid insurance policy in place; (b) whether the reported damage islikely to have occurred in a way and timeframe covered by the insurancepolicy; (c) whether the proposed repairs are acceptable under theinsurance policy or the criteria of the insurer; (d) whether the partprices quoted are within an acceptable threshold pricing for each part;(e) whether the proposed labour costs and time for repairs and/orpainting are appropriate for the determined damage to the vehicle; (f)whether each decision to repair or replace a part was correct; (g)whether the paint work on parts should be blended; (h) whether the partis an OEM or non-OEM part (i) whether a pre-painted part should orshouldn't be used; and (j) whether the correct option for a part hasbeen specified. Sometimes, the report to the insurer 1110 or the repairestimate provided to the insurer 1125 does not include sufficientinformation or detail when it is considered by the insurer 1130 and so,if the insurer determines that insufficient information or detail hasbeen provided 1135, the insurer requests further information 1140. Theinsurer can then reconsider the data provided 1130 again, make a similardetermination 1135 until sufficient detail is provided and then assessthe repair estimate 1145 and then reject the estimate 1150 or approvethe estimate 1155 so that the repair work can be performed 1160 by thegarage. If the estimate is rejected 1150, then the process is repeatedwhen the garage re-submits a revised repair estimate from step 1120.

Referring to FIG. 12, an example embodiment of a claim process 1200 withan incorporated automated repair cost estimate review process will nowbe described.

Following an accident or damage to a vehicle 1105, the insuredparty/client reports the accident to their insurer 1110 following theaccident 1105 and takes the vehicle to the body shop for a repairassessment 1115. The body shop prepares a repair estimate 1120 for thedamage to the vehicle and reports this estimate to the insurer 1125.

The data or information in relation to a particular claim is input intoan automated review system 1205, either manually by personnel at theinsurer entering it into the front end of the review system orautomatically by passing data received from the client and repair shopdirectly into the automated review system 1205.

In some embodiments the automated review system 1205 can carry outdifferent or a combination of tasks depending on the type of data inputinto the automated system 1205. For example, in the embodiment shown inFIG. 12, the automated review system 1205 triggers one of three reviewprocesses: a pre-defined workflow to be completed by a user 1210, theautomatic approval of claims with no determined anomalies 1220; or theflagging of cases where the automated system 1205 does not agree withthe repair estimate produced by the body shop 1230 so triggers a manual(i.e. human expert) or further review of such flagged cases 1235—and theexpert either agrees and the estimate is finalised 1225 or the expertdisagrees and the garage is required to prepare a revised estimate 1120.Thus, in this example embodiment, the automated review system 1205 canapprove, query or reject the body shop repair estimate in order tofinalise the repair estimate 1225 for approval.

In the example embodiment, of the automated review system 1205 is ableto perform a review by determining an independent repair estimate forthe claim being reviewed based on the claim inputs and using estimaticsproviders to determine an independent repair estimate using acomputer-generated damage estimate (generated from the claim inputs) andcomparing this independent repair estimate to the provided repairestimate from the garage 1125.

In example embodiments, workflows such as checking the VIN, checking ifrequired photos are missing, and checking that invoices match repairestimates can be performed. For example, one workflow can be triggeredwhen it is determined that there is an error or inconsistency with theVIN provided for the vehicle, or if the VIN hasn't been provided for thevehicle—the workflow can, instead of requiring the claim to be reviewedmanually, automatically return the repair estimate to the garage andrequest the VIN details for the vehicle before the repair estimate isconsidered for approval. Another example workflow would be triggeredshould it be determined that insufficient photos have been provided ofthe damage to the vehicle, and requesting further or better qualityphotos of the vehicle or specific areas of damage to the vehicle fromthe garage before the repair estimate is considered for approval. Afurther example workflow would be triggered should an invoice besubmitted for repair costs that do not match what was approved in aprevious repair estimate, either triggering a manual (human) review orsending an automatic rejection of the invoice to the garage providingdetails of the inconsistency with a previously submitted repairestimate.

In embodiments, the damage to the vehicle is determined from the photosof the damaged vehicle to determine the extent of the damage and theseverity of each piece of damage to the vehicle so it is important tohave sufficient image data of the vehicle and that this image data is ofsufficient quality. Some damage to a vehicle, such as structural damagecannot be determined from photos of the damaged vehicle, however.

With reference to FIG. 13, an overview of process 1300 performed by theautomated review system 1205 shown in FIG. 12 will now be described infurther detail.

The inputs 1305 provided to the automated assessment platform 1310include the information provided to the insurer by the client 1110 andthe repair estimate 1125 from the garage. In the example embodiment, theautomated assessment platform 1210 is provided with photographs 1320relevant to the claim (i.e. photographs of the vehicle, including forexample photographs of the vehicle from a few different angles aroundthe vehicle as well as close up photographs, generally of the damagedportions and parts of the vehicle), an order sheet 1325 prepared by theinsurer (i.e. details of the accident and/or damage prepared by theinsurer having spoken to the client) and the financial estimate providedby the repair shop 1330 (i.e. a list of the parts needing replacement,the materials required such as paint and a list of the labour requiredto remove and replace parts or repair parts and then perform finishingactions such as painting the repaired portions of the vehicle, theprices for the parts listed, the prices for any materials listed and theprices for any labour listed).

The automated assessment platform 1310 of the example embodiment hasthree layers: an ingestion layer 1345; an assessment engine layer 1365;and an application layer 1380. In example embodiments, the automatedassessment platform 1310 can assess the claim input through these layers1345, 1365, 1380 in sequence.

In the example embodiment, the ingestion layer 1345 handles the input ofdata into the automated assessment platform 1320. The ingestion layer1345 includes a standard application programming interface (API) 1335and a secure file transfer protocol upload module 1340. The standard API1335 provides a mechanism to input the claim input data 1305 into theplatform 1310, in this embodiment by being connected to an incommunication with the insurer systems into which the data has beeninput by a combination of the insured party, the garage and the insurerpersonnel. In the example embodiment, the insured party is provided witha web application on their mobile phone to take photographs of thedamage to their vehicle by the insurer and the image data captured usingthis web application is stored on the insurer computer systems andprovided by the standard API 1335 to the assessment platform 1310.Further, in the example embodiment, the body shop estimate 1330 areprovided to the insurer by the garage using standardised third partysoftware that captures the inputs from the garage personnel and preparesthe body shop estimate 1330 in a standardised format for the insurer anduploads this directly into the insurer computer systems using theinsurer's computer system API, which in turn provides this data to theassessment platform 1310 using the standard API 1335.

In the example embodiment, the assessment engine layer 1365 comprises ascope triage module 1350, a core assessment module 1355 and a workflowtriage module 1360. More detail on the assessment engine later 1365 willbe provided below with reference to FIG. 14.

In the example embodiment, the application layer 1380 comprises a lossadjuster application 1370 and a results API 1375. The loss adjusterapplication 1370 allows the insurer personnel to review the detailedindependent repair estimate that has been generated by the assessmentengine layer 1365 against the supplied garage repair estimate 1120, 1330to enable the insurer personnel to compare the details of each estimate,and to assist with efficient comparison the loss adjusted application1370 highlights any differences between the two estimates and providesan indication of the potential problems identified in the garageprovided body shop estimate 1330. For example, the body shop estimate1330 might list an additional labour operation that has not beendetermined to be required by the independent repair estimate generatedby the assessment engine layer 1365, and this would be highlighted viathe loss adjuster application 1370. The Results API 1375 allows for theoutput of the assessment engine layer 1365 to be sent to the relevantinsurer computer systems and personnel, for example to enable approvalof a repair estimate 1330 in the case where the independent repairestimate generated by the assessment engine layer 1365 concurs ormatches within a predetermined approval threshold set by the insurerpersonnel.

In an alternative embodiment, the supplied garage repair estimate 1120,1330 is used as an additional input by the assessment engine layer 1365which then outputs one or more determinations of incorrect, potentiallyfraudulent or inconsistent portions of the supplied garage repairestimate 1120, 1330.

In the example embodiment, in order to output an updated claim 1315 thatcan be or is approved by the insurer, the independently generated repairestimate from the assessment engine layer is output along with the claiminput data 1305 and any result output as well as any adjustments madevia the loss adjuster application 1330.

In some embodiments, the automated review (or assessment platform) 1310can include, however is not limited to, one or more layers, modules,platforms, interfaces and/or tools. For example, the automated reviewmay incorporate a platform 1310 to perform comprehensive automatedassessments of claims using models, standard APIs to integrate withclaims systems in the claim lifecycle and tools to help stakeholders usethese assessments in their workflows. In some embodiments, theassessment platform 1310 can have two layers: an ingestion layer 1345and an assessment engine layer 1365. In some embodiments, the ingestionlayer 1345 can comprise a standard Application Programming Interface(API) 1335 and an SSH File Transfer Protocol (SFTP) upload 1340 or anyother network protocol that may be used for secure file transfers. Insome embodiments, the assessment engine layer 1365 can comprise a scopetriage module 1350, a core assessment module 1355 and a workflow triagemodule 1360.

FIG. 14 shows a more detailed diagram 1400 of the assessment enginelayer 1365 shown in FIG. 13 of the example embodiment which will now bedescribed in more detail.

The claim data that is provided is assessed automatically using theautomated assessment platform 1310. As mentioned above, in the exampleembodiment, the assessment engine layer 1365 of the automated assessmentplatform 1310 comprises three major modules: the scope triage module1350, the core assessment module 1355 and the workflow triage module1360.

In the example embodiment, the scope triage module 1350 performs dataintegrity checks 1405 on the data received from the ingestion layer 1345by assessing claim photo coverage 1410, damaged panels photo coverage1415, and vehicle identification number (VIN) availability and validity1420.

The assessment of claim photo coverage 1410 determines, for thephotographs 1320 that have been provided to the insurer, whatpart/region of the vehicle is shown in each photograph. This allows anassessment of whether sufficient photos have been taken of the vehicle.

The assessment of damaged panels photo coverage 1415 then performsverification against the order sheet 1325 and the body shop estimate1330 that the photographs 1320 show all of the visible damage for whichrepairs, or replacements, are necessary in order for the automatedassessment platform 1310 to make an independent assessment of the damageto the vehicle and the likely repair work required using the photographs1320 in order to input these into estimatics platforms to generate anestimate. In some embodiments, if a predetermined threshold of coverageof the vehicle with the photographs 1320 provided is not met, then theprocess can terminate and a request be provided to provide additionalimages of the damaged vehicle, optionally indicating the parts/regionsfor which images are required. In other embodiments, the processcontinues and an assessment is made by the system 1400 for the coveragepossible with the images 1320 provided.

The determination of VIN (vehicle identification number) availabilityand validity 1420 checks that data is available for lookup for thesupplied VIN and that the VIN matches either the detected or providedfeatures of the vehicle.

In the example embodiment, the core assessment module 1355 is used tocreate an independent estimate of the repair work needed for the damagedvehicle based on the provided photographs 1320 that can then be used toverify the provided order sheet 1325 and body shop estimate 1330. Theindependent estimate of the repair work needed is generated by the coreassessment module 1355 and checked using a series of checking modules1430, 1435, 1440, 1445, 1450, 1455, 1460, 1465, 1470.

In the example embodiment, a leakage check module 1430 performs a checkto identify any aspects of the body shop estimate 1330 that seems eitherunnecessary or superfluous; or to identify any decisions of the bodyshop estimate 1330 that seem to be incorrect (e.g. an incorrect repairversus replace decision) using data from the insurer's records ofprevious approved and rejected claims. The leakage check module 1430 inthe present embodiment uses a repair replace check module 1435, a labourhours check module 1440, a paint check module 1445 and a parts checkmodule 1450 to check respective aspects of the order sheet 1325 and thebody shop estimate 1330.

The repair replace module 1435 checks that the repair and replace (i.e.that parts needing replacement and the parts needing repair rather thanreplacement) aspects of the order sheet 1325 and the body shop estimate1330 match the equivalent aspects of the generated independent estimateof the repair work needed for the damaged vehicle.

The labour hours module 1440 checks that the labour (i.e. the amount oflabour time and the labour operations/tasks to be carried out) aspectsof the order sheet 1325 and the body shop estimate 1330 match theequivalent aspects of the generated independent estimate of the repairwork needed for the damaged vehicle.

The paint module 1445 checks that the paint (i.e. the paint operationsand materials cost) aspects of the order sheet 1325 and the body shopestimate 1330 match the equivalent aspects of the generated independentestimate of the repair work needed for the damaged vehicle.

The parts module 1450 checks that the parts (i.e. the list of partsneeded) aspects of the order sheet 1325 and the body shop estimate 1330match the equivalent aspects of the generated independent estimate ofthe repair work needed for the damaged vehicle and that these aspectsare priced in line with data obtained from live data sources that can bequeried for part prices.

In the example embodiment, a fraud check module 1455 performs a check toidentify any aspects of the order sheet 1325 or the body shop estimate1330 that seem fraudulent. The fraud check module 1455 in the exampleembodiment uses an inconsistent claims check module 1460, aninconsistent damage check module 1465 and a vehicle ID check module1470.

The inconsistent claims check module 1460 checks for whether the ordersheet 1325 or the body shop estimate 1330 list repair work that has notbeen predicted from the input data in the generated independent estimateof the repair work needed for the damaged vehicle. For example, a claimmight be inconsistent if the reported point of impact is at the front ofa vehicle then small paint scratches to the rear of the vehicle areunlikely to have been caused by the impact and are more likely to havealready been present on the vehicle as damage prior to the impact.

The inconsistent damage check module 465 checks for whether the ordersheet 325 or the body shop estimate 330 list repair work that isexcessive compared to the repair work predicted to be necessary from theinput data in the generated independent estimate of the repair workneeded for the damaged vehicle. For example, damage might beinconsistent if a claim is made that a car had accidental damage occurovernight when it was parked but the assessed or reported damage hasmost or all of the characteristics of a vehicle that has been crashed byits driver.

The vehicle ID check module 1470 checks whether the VIN indicates thevehicle is the insured vehicle, that the vehicle is the correct make andmodel, and that the parts proposed to be used in the body shop estimate1330 are consistent with the make and model of vehicle retrieved fromlooking up the VIN in a third party database.

In some embodiments, for example, the checks may be implemented to checkagainst an independently generated repair estimate within higher andlower estimate thresholds for each of the modules 1430, 1435, 1440,1445, 1450, 1455, 1460, 1465, 1470. In other embodiments, only some ofthe check modules 1430, 1435, 1440, 1445, 1450, 1455, 1460, 1465, 1470are used to perform checks on the order sheet 1325 and body shopestimate 1330 against the independently generated repair estimate.

In the example embodiment, following the core assessment module 1355,the workflow triage module 1360 is then used to determine the output ofthe assessment engine layer 1365 or to determine what workflow istriggered as a result of the output of the assessment engine layer 1365.

The workflow triage module (or phase) 1460 in the example embodimentinclude assessment of usability of the output 1480, assessment of thevalue analysed 1485, determining a claim priority score 1490, and theapplication of any custom workflow rules 1495.

According to the example embodiment, the assessment of usability of theoutput 1480 is performed to assess whether the extent to which theoutput of the core assessment module (or phase) 1355 is of use. Thisassessment is performed using two modules: the value analysed assessmentmodule 1485 and the claim priority score module 1490.

According to the example embodiment, the value analysed assessmentmodule 1485 assesses what proportion of the body shop estimate 1330 havebeen checked against the independently generated repair estimate, as dueto the use of photos to generate the independent repair estimate it isnot possible to check all aspects of the repairs proposed to, forexample, internal structural damage not captured or detectable fromphotos of the damaged vehicle.

According to the example embodiment, the claim priority score module1490 assigns a priority to the claim based on the level of leakagedetermined 1430 and the level of fraud determined 1455. This can bedetermined using a predetermined scoring system or a relative scoringsystem versus other claims being processed substantially concurrently.

Further, according to the example embodiment, the is application of anycustom workflow rules 1495 that have been preconfigured by the insurerin order for the system to work with their internal policies, proceduresand systems.

In other embodiments, the output of the core assessment module 1355 isused to trigger certain workflows or actions. For example, should therebe no determined incorrect, potentially fraudulent, or inconsistentportions of the input claim then the input claim is automaticallyapproved. In other embodiments automatic approval is made if only aportion of the input claim under a predetermined threshold amount (forexample 20%, or 10% or 1% depending on insurer preferences) that isdetermined to be incorrect, potentially fraudulent, or inconsistent. Insome embodiments, if an incorrect VIN is detected in an input claim thena workflow can be triggered to avoid manual review of the claim andreturn the claim to enable the repair shop with a prompt to resubmit theclaim with the correct VIN data. In other embodiments, whereinsufficient data is determined to have been supplied by the repair shop(e.g. no VIN, a lack of photos of the vehicle or of one or more portionsof the vehicle) then a workflow can be triggered to again avoid manual(human) review and require the repair shop to submit the claim againwith the determined missing data. These automatically triggeredworkflows or actions can improve efficiency in processing claims byavoiding the needed for manual (human) review.

In some embodiments, incorporating the described automated reviewprocess can improve process of assessing proposed repair work in avariety of ways including improved damage assessment, the automation ofdamage assessment, the more efficient and/or prompt authorisation ofrepair work, and faster and more efficient settlement assessment. Damagecan be assessed more efficiently and accurately by triaging claims thatneed to be processed in different ways, based on for example the valueof the claim that can be analysed by the system automatically and/or thepriority of the claim. The use of automated assessment can acceleratefinding leakage in claims while allowing the quick approve claims withno anomalies automatically. The automated review process of otherembodiments can be used to assess and flag post-repair leakage byreviewing approved invoices and performing post repair audits. In someembodiments, the various assessments can be used for triaging claims todetermine whether the best way to process a claim is to send an experton-site if structural damage has been identified, partial processing toquickly identify leakage or fraud on a claim for insurer personnel toreview and challenge, and automation to automate approval of low damageclaims or high certainty claims based on insurer predeterminedstatistical or absolute thresholds set for damage or leakage forexample.

FIG. 15 shows a comparison analysis process 1500 as implemented in oneexample embodiment where the process is used to perform the checking ofthe images of the damage to the vehicle and the repair estimatesubmitted by the repair shop using a model trained on historical data.

Specifically, FIG. 15 shows a flowchart depicting a method 1500 forassessing proposed vehicle repair estimate data against historical dataand predetermined approval criteria by processing image input 1505 ofthe damage to the vehicle to be repaired using an image classifier 1510.In this embodiment, the vehicle repair estimate data is analysed using a(decision) classifier 1515, which has been trained using historicalvehicle repair data and works in conjunction with an estimatics platform1535 by determining the damage to the vehicle and uses the estimaticsplatform to determine repair data for the determined damage in order toverify the input vehicle estimate data 1520, to output any identifiederrors or anomalies 1525 in the input vehicle estimate data 1520. Insome embodiments, the input vehicle estimate data 1520 along with anyerrors or anomalies 1525 can be displayed to a user in a user interfaceon a computer for review, for example in a loss adjuster application.

FIG. 16 shows a comparison analysis process 1600 as implemented inanother example embodiment where the process is used to perform thechecking of the images of the damage of the vehicle against historicaldata by generating an independent repair estimate 1615 and comparingthis to the submitted repair estimate 1620 from the repair shop.

Specifically, FIG. 16 shows a flowchart depicting a method 1600 forassessing proposed vehicle repair estimate data 1620 against historicaldata and predetermined approval criteria by processing images of thedamage to the vehicle 1605 using an image classifier 1510. In thisexample embodiment, an independent repair estimate is generated by the(estimate) classifier 1615 in conjunction with lookups made to anestimatics platform 1635, where the image and classifiers 1610, 1615have been trained using historical vehicle repair data and work inconjunction with an estimatics platform 1635, to output an independentrepair estimate. The independent repair estimate and the repair estimate1620 provided by the repair shop are compared 1625 to identify errors oranomalies 1630 in the input vehicle estimate data 1620. In someembodiments, the independent repair estimate and input vehicle estimatedata 1620 along with any errors or anomalies 1630 can be displayedside-by-side to a user in a user interface on a computer for review, forexample in a loss adjuster application.

Example embodiments, such as the example embodiment process 1700 shownin FIG. 17, can be used by body shops to enable them to check theirproposed repair estimate and, if any irregular operations areidentified, change their proposed repair estimate to remove anyirregular operations or improve the supporting evidence for necessaryoperations flagged as irregular before the repair estimate is sent tothe insurer, in order to comply with the requirements of automatedassessment of their repair estimate by the insurer using the automatedassessment platform.

By previewing the claim estimate, and the associated automatedassessment of the claim estimate, the body shop can be made aware ofpotentially incorrect decisions used to generate their repair estimateor whether they will be required to submit more information (such asphotographs to support their proposed repairs, for example). In someembodiments, the system/platform can be used to produce an estimateautomatically for the body shop by analysing photographic images of thedamaged vehicle obtained by the body shop and obtaining repair datausing estimatics providers.

In the embodiment shown in FIG. 17, a process for pre-checking a repairestimate 1700 is shown and will now be described. When an accidentoccurs 1705, the client reports the accident and details of the accidentand the damage to the insurer 1710. The client takes their damagedvehicle to a garage for repair 1715. The garage surveys the damage andtakes photographs of the damaged vehicle, and inputs these photographsand some details of the vehicle and damage/accident into the estimateplatform 1720. The estimate platform used a model trained on car damagedata to prepare a determination of the damage to the vehicle and theseverity of this damage, including for example whether to replace orrepair parts that have been damaged. This determination is submitted toan estimatics platform to determine for example the parts list requiredand the labour hours and costs in order to prepare the repair estimate.The repair estimate is then provided to the insurer 1725. Since theplatform has already prepared the repair estimate using the verifiedprocess for assessing the damage to the vehicle, the repair estimate caneither be pre-approved or can be verified by the insurer software in theway used for manually prepared repair estimates. Once approved, thegarage can begin work.

Optionally, should the estimate have been prepared incorrectly, or awarning message is overridden or ignored by the user and so a claim issubmitted that isn't pre-approved, it is possible for the insurer toprocess the estimate 1730 and reject it, returning the process to thestep of the garage preparing an estimate 1720 for resubmission 1725.

Referring now to FIGS. 18 to 20, several embodiments illustrating one ormore further aspects will now be described.

Referring first to FIG. 18, a generalised embodiment 1800 will now bedescribed in more detail to provide an overview of the approach taken bythe methods of various aspect/embodiments.

Images 1805 are captured of a damaged vehicle, typically these are aplurality of images of the vehicle from a variety of viewpoints of thevehicle and typically these show the damage to the vehicle both close-upand also in context with undamaged portions of the vehicle. These imagesare typically captured as part of the process of making an insuranceclaim, but may also be captured by an occupant or owner of the vehicleshortly after the damage occurs or by a vehicle repair business as partof documenting their damage assessment.

The images 1805 are provided to a plurality of image classifiers 1815,each classifier 1815 outputting a decision on whether each of the images1805 is classified as containing any damage to a specific normalisedportion of the vehicle (e.g. the front wing; the front left door; therear bumper; etc). In other embodiments, a single classifier 1815 may beused to determine damage classifications to more than one portion of thevehicle or the entire vehicle.

The output of the image classifiers 1815 can be used a set of decisions(or in some embodiments classifications, optionally with confidencevalues), each being a decision per portion of the vehicle, as to whetherthere is damage to that portion of the vehicle. In some embodiments, alevel of damage is also indicated in the decision that is output fromthe image classifiers 1815 (e.g. minor or major; or damage categoriessuch as retouch, repair or replace). In some embodiments, one or morelocations of damage are indicated in the decision that is output fromthe image classifiers 1815. In some embodiments, other features are alsooutput from the image classifiers 1815.

In addition to the images 1805, a claim input 1810 is provided in theform of an electronically generated detailed list of parts and labouroperations to perform a repair to the damaged vehicle. The claim input1810 is previously generated using a third party database, provided withinputs by vehicle repair personnel based on their assessment of thedamage to the vehicle, which generates an output list of parts for thespecific model of vehicle and repair operations required to repair orreplace each of the damaged components of the vehicle for the vehiclerepair personnel.

The claim input 1810 is provided to a NLP (natural language processing)model 1825 which reads the human readable text in the claim model 1810and converts the claim input 1810 into normalised data for each portionof the vehicle (for example, converting a portion of the claim inputcomprising a detailed list of ten parts and corresponding list of manualoperations and associated labour times into normalised data such as“replace front left wing, three hours total labour time, requires Xparts” where X could be a financial value, a list of specific parts or anumber of parts). In some embodiments, any associated photos from theclaim input 1810 are included in the normalised claim input generated bythe NLP model 1825. In embodiments, the NLP model 1825 comprises aplurality of NLP models.

Natural Language Processing (NLP) is a field of machine learning where acomputer is able to understand, analyse, manipulate and generate humanlanguage. The NLP model 1825 uses techniques from this field tounderstand the human readable text in the claim model 1810 and convertthis into a standardised set of information that is used to train and atruntime with the machine learned models of the embodiment. Building amapping and parsing system using NLP can allow the method of this andother embodiments to correctly ingest estimates from third partyestimatics systems, being known or new estimatics systems, with zerotraining data, even in new markets/jurisdictions. In some embodiments, acharacter-level language model is used to predict based on a partdescription which can then be mapped to a normalised portion of thevehicle (or normalised part), for example a front bumper. Inembodiments, the language model is trained on historical unlabelledestimate data and/or across different languages—this can make thelearning process more efficient and/or increase performance at inferenceand/or allow the model(s) to work better when applied to newgeographies/markets/jurisdictions.

The decision(s)/classification(s) from the image classifiers 1815 andthe normalised claim input from the NLP model(s) 1825 are provided to aclassifier 1820. The classifier is a trained model or models thatassesses the output from the NLP model(s) 1825, which represents therepair estimate generated by the vehicle repair personnel, to the outputfrom the image classifier(s) 1815, which represents an independentlygenerated damage estimate (and in some embodiments the method can useone or more third party databases to generate a detailed list of partsand labour operations based on the independent generated damage estimateoutput from the image classifier(s) 1815), in order to classify whetherthe claim input 1810 (in whole or in part) is correct.

The output of the classifier(s) 1820 can be used to verify a repairestimate generated by the vehicle repair personnel, for example by thevehicle repair personnel before it is submitted for approval by acustomer or an insurer (to ensure it is correct) or by an insurer (toensure it is correct to approve or reject a claim for the correct repairwork from an insured customer, for example as part of an insurance claimapproval process).

To provide more detail on the concept of normalised car parts 1900, asused in above described example embodiment, the following detail isprovided with reference to FIG. 19.

In FIG. 19, there is shown a generic model of an example vehicle 1910.The example vehicle 1910 shows a selection of normalised parts/regions1900 that are common to most if not all vehicles, which commonnormalised parts 1900 can be used to describe in generic terms thetypical parts found of most vehicles that might need to be considered bythe method for assessing or verifying estimates for repair work.

For example, the example vehicle 1910 has shown a front right wing 1920,a front left wing 1930, a left front door 1940 and a left back door1950. Other possible normalised vehicle parts 1900 might include(non-exhaustively): bumpers, wings, wheel arches, roofs, windows,wheels, lights, grills, vents, radiators, mirrors, windscreens, andconvertible roofs.

Using this normalised part schema allows the conversion of detailedrepair estimates provided by vehicle repair personnel into normalisedrepair inputs, for example to covert a detailed part list and detailedlabour times and operations into a normalised indication, for eachnormalised portion of the vehicle, of what damage is to the vehicle ofinterest and a summary of the parts and labour required to repair eachnormalised portion of the vehicle.

Similarity, using the normalised part schema, an analysis can beperformed on the image data of the vehicle to assess which of thenormalised parts of the vehicle exhibit damage in the photos of thevehicle and, in some embodiments, the extent (e.g. entire part/majorityof part/minority of part) and/or severity (e.g. high/medium/low) and/orclassification (e.g. replace/repair/paint) and/or location (e.g.position or positions on the vehicle) and/or other features of thedamage to each normalised portion.

With reference now to FIG. 20, a more detailed example embodiment of theindependent damage generation process 2000 will now be described morefully.

The input to the process are a plurality of images 2005, which comprisesa set of images of the damaged vehicle from different viewpoints andincluding image data of the vehicle as a whole and image data pertainingto the damage to the vehicle.

The images 2005 are provided to a Visual Ai model 2015, in this exampleembodiment a machine learned model trained on historic appraiserdecision data which includes image data of the damaged vehicles and thecorresponding appraiser decisions for the damage on these damagedvehicles. Other machine learned and artificial intelligence approachescan be employed in other embodiments to provide a Visual Ai model 2015.The Visual Ai model 2015 outputs a damage mask 2010 and a damageclassification for each normalised portion of the damaged vehicle.

In some embodiments, using the normalised part schema/arrangement, anddetermining the extent, severity, location and type of damage can allowfor the use of models or look-up tables (e.g. using third partydatabases or by generating look-up tables) to determine a more detailedlevel of granularity in output determinations (e.g. specifying the exactpart number for a damage bumper for the make, model and year of vehiclebeing considered and how long it will take to replace the part). Thusthe benefit of a normalised part schema can be that it allows ageneralised visual assessment across many types of vehicles and/orgeographies (allowing the visual modelling to be performed more easilyand/or robustly) without losing the ability to determine a detailedoutput (e.g. specific parts and/or operations and/or labour timerequirement).

The damage mask 2010 comprises a damage classification for allnormalised parts of the damaged vehicle.

The damage classification for each normalised portion of the damagedvehicle is output from the Visual Ai model to the repair model 2020.

The repair model 2020 comprises a set of models including a set ofrepair/replace models and a repair labour hours model, all of which aremachine learned models trained on historic appraiser decision data whichincludes image data of the damaged vehicles and the correspondingappraiser decisions for the damage to these damaged vehicles. Othermachine learned and artificial intelligence approaches can be employedin other embodiments to provide a repair model 2020 and/orrepair/replace models and/or one or more repair labour hours models. Theoutput of the repair model 2020 is a classification, for each of thenormalised parts of the vehicle, of whether the normalised portion ofthe vehicle requires repairing or replacing and the labour operationcategory (e.g. minimal/straightforward/complex or a rough estimate oftime required).

A blend engine 2025 receives both the damage mask 2010 and the output ofthe repair model 2020. The blend engine 2025 then assesses whether anypaint blending will be required for each of the normalised ports of thevehicle. Paint blending is the process of ensuring a smooth transitionin paint colour between neighbouring parts of a vehicle, thus may berequired (for either a repaired/replaced part and/or at one or moreneighbouring undamaged parts) to ensure a contiguous finish to the paintcolour if a pre-painted replacement part is used to replace a damagedpart or if a damaged part is painted following repair or replacement.The output of the blend engine 2025 is provided to the paint rulesengine 2030 and the remove and install (R&I) rules engine 2035.

The paint rules engine 2035 comprises a set of rules used to assess theoutput of the blend engine and the damage mask 2010 to estimate thelikely materials and labour required to paint the repaired vehicle. Theset of rules in the example embodiment are hand crafted but in otherembodiments a machine learned approach can be used.

The R&I rules engine 2035 determines which parts need to be removed fromthe vehicle so that the physical repair operations can be carried outand then re-fitted. In some embodiments, the output from the R&I rulesengine 2035 is an aggregate amount of time that needs to be spent onremoving these parts from the vehicle and re-fitting these parts oncerepairs have been made, per normalised part of the vehicle.

Based on the outputs of the Visual Ai model 2015, the repair model 2020,the blend engine 2025, the paint rules engine 2030 and the R&I rulesengine a database lookup 2040 is performed at a 3^(rd) party databased2050 to generate an electronically generated detailed list of parts andlabour operations to perform a repair to the damaged vehicle, using theassessment generated of the damage to the vehicle and details about thevehicle (e.g. make, model, year, etc if any or all of this data isavailable).

The output of the damage assessment process 2000, once the detailed listhas been prepared using the database lookup 2040, is reviewed by a setof overlap rules at an overlap rules engine 2060 to determine if anyduplicate parts or labour operations are listed in the detailed list,which can then be removed from the detailed list by the overlap rulesengine 360 and an output produced 2070.

The output 2070 can then be compared to an electronically generateddetailed list of parts and labour operations to perform a repair to thedamaged vehicle prepared by vehicle repair personnel.

By breaking up the process of generating a repair estimate into humandecisions and software-assisted decisions, and building machine learnedmodels to predict the human decisions, a completely automated approachto generating a repair estimate can be adopted.

In some embodiments, some of the models used can have a limited capacityto prevent overfitting to the training data, for example by applying arule-based system or using a shallow, capacity constrained machinelearning model/approach.

In some embodiments, one or more of the models are trained in a manneragnostic to the make, model and year of the vehicles being considered.This can allow the training of a universal set of one or more models, oreven a universal approach, to generating classifications ofdamage/damage estimates for vehicles or a generalisable damagerepresentation (from, for example, just image data of the damagedvehicle). In embodiments, some or all of the models comprise a finalshallow layer which takes the generalisable damage representation andadds the make, model and year information for the vehicle beingconsidered in order to make the final prediction/classification. Inthese embodiments using the final shallow layer to add the make, modeland year information, the approach can require much less training datato train the model(s) that a full convolutional neural network.

In some embodiments, the model(s) are trained using data from multiplegeographies, such that the first layers of the neural networks in themodel(s) processes the input data in a universal way across allgeographies in order to extract generalisable features for vehicles,then each geography is refined using only training data from thatgeography in order to learn the specific repair methodologies for eachgeography. In embodiments, the training dataset can be pseudo-labelledwith geography specific labels so that the geography specific branchesof the model can be treated as branches in a multi-task learningtechnique. In other embodiments, one or more models can be trained tore-weigh the loss function to account for the proportion ofrepair/replace decisions in each geography. In other embodiments, one ormore models can additionally have a confusion loss layer present insubstantially the middle of the network, which can be used to enforcethat at that point in the network it doesn't differentiate betweenimages coming from different geographies.

In some embodiments, the method normalises the point or impact (e.g. thefront left of the vehicle) and/or the object involved in the collision(e.g. a tree, or another car) and includes a further step of checkingthat the damage prediction/classification is consistent with either orboth of the point of impact and the object involved in the collision.

An alternative embodiment with now be described in more detail where noprevious data has been acquired in respect of a new situation, forexample where a new make, model and year of a vehicle is encountered.

In this embodiment, upon encountering a new make, model and year of avehicle, the system reverts to using a “default” vehicle type (or, inother embodiments, the system may try to determine a generic car typesuch as a “sedan”, “pick-up” or “van” and use pre-determined orpre-learned data for that vehicle type). The system uses either the“default” or the selected “generic” vehicle type at every subsequentstep where otherwise a specific make, model and year is used (forexample as input to a final layer of the visual models, or as an inputto the blend engine and/or paint rules and/or R&I rules and as inputwhen querying the third party database).

By training a generic model that uses normalised parts per vehicle, newmodels of vehicle can be analysed immediately (using the new make/modelas an input) and the machine learned models can learn correlations withexisting learned parameters for other make/models. In this embodiment, adefault setting for the make and model of a vehicle is used, having beentrained on training data lacking make, model or year information,allowing “zero-shot” prediction for the damage representation for avehicle of unknown make, model and year (but with an expected loweraccuracy than if at least some training had been performed specificallyfor the make, model and year).

Another embodiment will now be described in more detail where a trainedapproach in one jurisdiction is adapted to a different unseenjurisdiction.

In such an embodiment, the system is configured as in the previousembodiment described in relation to FIG. 20, but the design decisionsmade allow the method to be adapted to a new geography with little data.A relatively small amount of data is collected in respect of thedifferent unseen jurisdiction (i.e. new data for the new jurisdiction).In particular, by finetuning just the last layers of the visual models(which requires a lot less data than training a new visual model, hencethe small amount of data collected for the new jurisdiction), using thisnew data and by adjusting the blend, paint and R&I engines based on thenew data (which also requires much less data than training machinelearning models from scratch to perform these operations), thepreviously-described approach(es) can be adapted to a new domain (i.e.to the different unseen jurisdiction).

By using domain adaption, a model can be trained to work universally butthen adapted to be used with different markets/jurisdictions/models. Insome embodiments, a model is generated for one market (for example, theinsurance repairs market in France) and then, using domain adaption,fine-tuned for use in another market (for example, the insurance repairsmarket in Japan). For example, the initial dataset used to train themodels may have a very large amount of data compared to the followingdataset (using the previous example, there might be one hundred timesthe data that can be used to train the model for France than for Japan).The differences between markets/jurisdictions/models may be any numberof factors, for example in some jurisdictions it is typical to replaceparts rather than repair them. This approach can also allow a high levelof performance when adapting to local changes (e.g. a hailstorm or othernatural disaster has caused widespread damage) by adjusting the priordistribution of labels. In some embodiments of this approach, live keymetrics are tracked, including for example metrics such as theproportion of normalised parts that are repaired vs those which arereplaced. When a drastic shift in distribution of tracked metricsoccurs, it would indicate something significant has changed in howvehicles are getting damaged and/or are being repaired or proposed to berepaired (for example, a hail storm or natural disaster). Usingthresholding, the approach embodied in the system can be adjustedaccordingly (i.e. when the distribution has changed by over apredetermined amount) by tuning any one or more of the blend/paint/R&Iengines or by fine-tuning the top layers of the visual classifiers inorder to allow adaption to changes in repair methodologies quickly.

In some embodiments, an efficient categorial feature encoding is usedfor the make and model in order to further reduce the amount of trainingdata needed.

Referring now to FIG. 21, an alternative embodiment implementing a moredetailed set of normalised vehicle parts 2100 (ornormalised/standardised vehicle parts or regions) will now be describedin more detail. This more detailed set of normalised vehicle parts 2100can be used with any of the other described aspects and/or embodimentsherein.

Shown in FIG. 21 is a generic vehicle diagram according to anembodiment, where the parts/regions of the vehicle are broken up into aplurality of normalised/standardised parts, zones or regions 2100. InFIG. 21, there are shown the following normalised/standardised parts: afront left headlight 2102; a front radiator grille 2104; a front rightheadlight 2106; a front bonnet 2108; a front right fender 2110; a frontright wheel 2112; a right running board 2114; a front right door 2116; afront right door window 2118; a back right door 2120, a back right doorwindow 2122; a trunk 2124; a rear right quarter panel 2126; a rear rightlight 2128; a rear bumper 2130; a rear left light 2132; a read leftquarter panel 2134; a rear window 2136; a back right door window 2138; aback right door 2140; a left running board 2142; a front left door 2144;a front left door window 2146; a front window 2148; a front left wheel2150; a front left fender 2152; and a front bumper 2154. In otherembodiments only some of these normalised parts may be used and/or someother normalised parts may be exchanged for those used in the exampleembodiment.

Referring now to FIG. 22, a paint check process 2200 according to anexample embodiment is shown and will now be described in more detail.

In this embodiment, the input data is a plurality of photos 2202 of thedamaged vehicle and a repair shop estimate 2204 which includes detailsof the proposed repair operations and/or materials required to repairthe damage to the damaged vehicle (typically along with the proposedcost for the proposed repair operations and/or materials).

Initially, some pre-check checks 2206 are performed. The pre-checkchecks 2206 include (a) determining whether the claim is eligible, i.e.determining whether the proposed repair operations and/or materials arecovered by an insurance policy; (b) determining whether the proposedrepair operations including proposed painting of an in-scope and/orout-of-scope panels of the damaged vehicle; and (c) to determine whetherthere are look-up tables for the specific make, model and year of thedamaged vehicle in order for the system to be able to retrieve therelevant data required to determine the paint requirements for thedamaged vehicle. In other embodiments, only some of these checks areperformed and/or alternative or modified checks may be performed.

Following the pre-check checks 2206, the paint area is determined 2208.Specifically, the areas of the damaged vehicle that require painting aredetermined. To do this, per-part models are used to determine whethereach of the parts of the vehicle require painting and to whatextent—i.e. does only a spot on a panel need to be painted, or does aportion of the panel need to be painted, or does most/all of the panelneed to be painted. This is described in more detail below in relationto FIG. 23 and the underlying models used in the process 2200 can beadapted to different jurisdictions by for example applying differenttraining weights. Adapting models to new jurisdictions can be done invarious ways. In one embodiment, it can be done by using domain adaptionmethods with data from the new jurisdiction. In another embodiment, itcan be done by adjusting the weight of each class (e.g. spot vs minor vsmajor, as described elsewhere) based on the prevalence in thatjurisdiction. In a further embodiment, it can be done by settingdifferent thresholds on the output of the classifier to determine whatlevel of damage corresponds to each class class (e.g. spot vs minor vsmajor, as described elsewhere).

Once the paint area has been determined, the paint time is determined2210—specifically an estimate for the time it will take to paint thedetermined paint area is generated. To do this, in this embodiment, alook up is performed (i.e. a query sent) to a third party database isperformed for all of the parts/panels determined to require painting,including details of the extent to which each of the panels requirepainting, and the third party database returns a time value based on thequery, typically the time value based on the specific make, model andyear of vehicle.

In parallel with the paint area determination process 2208, apre-painted panel check 2212 is performed. The pre-painted panel check2212 determines whether the damage to the damaged vehicle will requireany pre-painted panels to be used to repair the damaged vehicle, thusremoving the need to paint one or more parts of the vehicle due to theparts used to repair the vehicle being pre-painted.

For the pre-painted panels determined to be required by the pre-paintedpanel check 2212, two further processes are performed: a determinationof the replace paint time 2214; and a determination of the addition cost2216.

The determination of the replace paint time process 2214 comprisesdetermining the time required to perform the replacement of the damagedpanel(s) with the pre-painted panel(s). To do this, in this embodiment,a look up is performed (i.e. one or more queries are sent) to a thirdparty database for all of the parts/panels determined to requirereplacement with a pre-painted panel, and the third party databasereturns a time value based on the query, typically the time value basedon the specific make, model and year of vehicle.

The determination of the addition cost process 2216 comprisesdetermining the requires auxiliary task that need to be performed inorder to replace any parts with pre-painted parts or prepare the damagedvehicle for painting the areas determined to require painting,including: determining the time required to apply masking tape toprevent paint coating areas of the car not to be painted; and to preparethe paint to be applied (e.g. to mix the paint).

A total paint time is then determined by combining the determined repairpaint time from the paint time determination process 2210, the additioncost process 2216 and the determine replace paint time process 2214. Insome embodiments, overlap between the times can be subtracted from thetotal time determined by the process 2218.

Following this, a paint labour cost is determined 2220, using the totaltime determined 2218 and a lookup performed using a third-party databaseto obtain appropriate labour rates for the time required to perform thepainting operations.

Following this, a material cost is determined 2222. Specifically, thecost of the paint required for the paint areas 2208 is determined fromby adding together the materials cost of the paint and the pre-paintedpanels required. The cost of the materials can be determined for themake, model and year of the damaged vehicle by performing a third partydatabase lookup to retrieve the values required.

A total paint cost is then determined 2224 by combining the labour cost2220 and the material cost 2224.

Then some post-check checks 2226 are performed, including in thisembodiment: checking that unnecessary parts of the vehicle aren't beingproposed to be painted (e.g. the door frame or the inside of a panel),and/or including items inconsistently (e.g. as materials but not labourproposed), in the proposed repair operations and/or materials.

Finally, one or more decisions as to whether the proposed repair shopestimate 2204 contains any unnecessary or anomalous proposed repairoperations and/or materials is generated based on the pre-check checks2206, the post-check checks 2226 and a comparison of the determinedtotal paint cost 2224, the determined material cost 2222, and the paintlabour cost 2220 to the proposed repair operations and/or materials inthe repair shop estimate 2204.

As shown in FIG. 23, it should be noted that differentjurisdictions/geographies have different thresholds, preferences, schemaand rules that impact what is acceptable in proposed repair operationsand/or materials used to repair damaged vehicle—an example of potentialdifferences between jurisdictions 2320 is shown in table 2300 and willnow be described in more detail below.

In table 2300, there are shown five generic example jurisdictions A, B,C, D and E. Each of the jurisdictions have different rules 2322, 2324,2326, 2328, 2330 for what to do per assessment of paint requirements2330. The “AI decision”, i.e. the decision/classification of theper-part model assessing whether a part of a damaged vehicle requiresrepainting in this example embodiment can output four classifications:none; spot; minor and major. However, to adapt these classifications tothe relevant jurisdictions, the jurisdiction specific rules 2300 need tobe selected as appropriate. So for example, in jurisdiction A the rules2322 specify that a spot paint job requires a fifth of the panel to bepainted while a minor paint job requires half of the panel to be paintedand a major paint job requires the whole panel to be painted. Incontrast, in jurisdiction B the rules 2324 specify that a spot paint jobrequires just the spot of the panel to be painted while a minor paintjob requires less than fifty percent of the panel to be painted and amajor paint job requires more than fifty percent of panel to be painted.Further, in jurisdiction C the rules 2326 specify that a spot paint jobrequires just the spot of the panel to be painted while a minor paintjob requires twenty percent or less of the panel to be painted and amajor paint job requires more than twenty percent of the panel to bepainted. In further contrast, in jurisdiction D the rules 2328 specifythat a spot paint job requires just the spot of the panel to be paintedwhile a minor paint job requires the panel to be painted at a cost fiftypercent below fully repainting the panel and/or replacing the panel witha pre-painted panel and a major paint job requires the panel to bepainted at a cost twenty percent below fully repainting the panel and/orreplacing the panel with a pre-painted panel. Finally, in jurisdiction Ethe rules 2330 specify that a spot paint job requires thirty percent ofthe panel to be painted while a minor paint job requires sixty percentof the panel to be painted and a major paint job requires ninety percentof the panel to be painted.

Referring now to FIG. 24, to implement the models in the aboveembodiments, in an example embodiment, a model architecture 2400 is usedincluding the use of a plurality of gradient booster trees 2440 whichwill now be described in more detail below.

Specifically, the model architecture 2400 used is a two-layerarchitecture using both visual models 2420 and gradient booster trees2440.

The input to the visual models 2420 are the photos of the damagedvehicle 2410 and the repair shop estimate 2430. The visual models 2420are trained on a per-part basis to determine, for each part of thedamaged vehicle, whether and what degree of painting is required: none;spot; minor or major. These visual models are run on photos of all partsof the damaged vehicle (as shown in the photos/images). In addition, inthis embodiment, visual models are also run to determine: (a) whether areplace or repair operation is required for each panel; (b) the amountof repair labour is required for each panel; (c) whether each part isdamaged or undamaged. All of the scores from these visual models(painting; repair/replace; labour; damaged/undamaged) are input into thegradient booster trees 2440, across multiple parts, and the output is aprediction of a final score. The purpose of the gradient booster trees2440 are to take information from the multiple visual models, and frommore than one panel (e.g. just two up to all of the panels on thevehicle) and use this information to make a final paint determination inorder to improve on just using a visual model 2420 to determine paintrequirements per panel and combine the scores. The gradient booster tree2440 comprises multiple decision trees having multiple nodes 2450 a, b,c, d, e, f, g and multiple terminal nodes 2460 a, b, c, d, e, f, g, h.Each individual decision tree can be quite a weak predictor by itself,but the boosting algorithm used can concentrate each one on a region ofthe prediction space such that, in aggregate, they form a strongpredictor of the damage. In this embodiment, this is done by addingtrees in an iterative fashion, using an algorithm such as AdaBoost,details of which are herein incorporated by reference.

Referring now to FIG. 25, an equation for calculating paint cost 2500according to an embodiment is presented and will now be described inmore detail below.

The total paint cost 2510 is shown to comprise the sum of multiplefunctions including the labour rate 2520, the material cost rate 2530,the paint type and vehicle details 2540, the repair paint times 2560,the total repair paint time 2550, the total replace paint time 2570, therepair paint times 2580 and the addition paint time 2590.

Referring now to FIG. 26, an overview of the damage determinationprocess 2600 according to an embodiment will now be described in moredetail below.

As an input to the damage determination process 2600, the photos of thedamaged vehicle 2610 are provided. In this process, there is a need todetermine if a normalised panel is undamaged or if there is some damagepresent. The photos 2610 are provided to a visual damage classifierwhich provides classification of the damage to each part of the vehiclefor each of the photos using classifiers trained per part that classifyeach of the photos 2610. Following this, a process of image segmentationis performed 2630 to identify segments of interest, or to segment damagewithin the images. Both the output of the visual damage classifier 2620and the segmentation 2630, along with the photos 2610, are then used tooutput a final damage score 2650 using the damage classifier 2650. Thiscan allow the final damage classifier 2640 to use the segmentation mask2630 with the photos 2610 to better focus on and/or assess the segmenteddamage shown in the photos. In some embodiments, the segmentation 2630outputs a segmentation mask indicating where it is most likely damagewill be determined to have occurred from the photos 2610. In someembodiments, the visual damage classifier 2620 may output with very lowcertainty where damage has been determined in the photos 2610.

Referring now to FIG. 27, an alternative embodiment of a damagedetermination process 2700 is shown and will now be described in moredetail.

Specifically, this embodiment is concerned with damage such as cosmeticdamage to a vehicle, which can be very hard to detect in images of thedamaged vehicle, especially where for example the vehicle has a verylight-colored paint. Typically, if the images of the damaged vehicle aredown-sampled during the inference process by the models, it is possiblethat such damage becomes difficult or impossible to determine. In theprocess 2700, the photos 2710 that are input are cropped 2715 in arandom fashion to generate multiple cropped images for each image. Inother embodiments, a part classifier can be used instead of performingrandom cropping in order to select particular portions of the vehiclethat are likely to contain the part(s) of interest). The cropped imagesare then processed by the visual damage classifier 2720 to determine anydamage on a per-part basis for each cropped image. The cropped images2715 and the original images 2710 as well as the output of the visualdamage classifier 2720 are provided to the image segmentation process2730 as per the embodiment of FIG. 26 and the damage classifier 2740outputs a final damage score 2750.

Referring now to FIG. 28, further details of the segmentation 2800 willnow be shown and described in more detail.

The figure shows an example vehicle 2810 as might be shown in an imageof a damaged vehicle, as well as a cropped portion of the image 2840 inwhich damage 2850 is more clearly visible and which can be segmentedusing a bounding box (or similar technique) 2860. Segmentation data canbe collected, for example from human labellers creating training data,and a segmentation model trained on this data to perform segmentation ofimages (cropped or otherwise) to identify damage or regions of interestin the images.

In some situations, the images input will appear to show no damage inmost of the images but only in the close-up images can the damage beenseen. To output a substantially accurate score, the models need to beable to determine that the part is damaged in spite of the damage notbeing visible in most of the images of the part and in addition themodel needs to be able to locate the damage visible in the close-upimages to a location on the vehicle, which can sometimes be difficult asthere is limited context to close-up images.

The segmentation performed on the image data can then be used by thedamage classifiers to improve the ability to identify otherwisedifficult to detect damage to vehicles, for example cosmetic damage thatwould otherwise only comprise of a very small portion of an uncroppedimage and which might only be visible in a close-up image. Thus,segmenting the area on the vehicle where the damage is located canassist the models to determine both that the damage is present and alsowhere the damage is on the vehicle, as the damage might be identifiablein a close-up image of the vehicle but due to the close-up nature of theimage the model may be unable to locate the damage on the vehicle.

Referring now to FIG. 29, an overview of the damage determinationprocess 2900 according to an embodiment will now be described in moredetail below.

Multiple images 2910 are input into the process 2900, each of which iscropped into multiple crops of each image 2920′, 2920″, 2920′″, 2920 n.Each of the multiple crops of each image 2920′, 2920″, 2920′″, 2920 nare then processed by a multi-image model 2930. In this embodiment, themulti-image model 2930 comprises a visual damage classifier 2940,followed by a pooling operator 2950 then a final damage classifier 2960,which outputs a final damaged/undamaged score 2970 for each part.

By cropping the multiple images 2910, the image resolution of the imagesare preserved in the cropped images 2920′, 2920″, 2920′″, 2920 n so thatthe full detail of the original resolution of the images can beprocessed by the multi-image model 2930 without needing to down-samplethe images 2910. In alternative embodiments, instead of cropping theimages 2910, the model 2930 can be trained to work at the originalresolution of the input images without down-sampling.

By training and using a multi-image model 2930, which only outputs aprediction score of whether each part is damaged or undamaged 2970 afterlooking at all of the images 2910, the model 2930 can substantiallyovercome the limitations in the input data only showing damage clearlyin some images of the vehicle and not in others. The multi-image model2930 achieves this by the visual damage classifier 2940 producing afeature representation for each of the images 2910 and/or multiplecropped images 2920′, 2920″, 2920′″, 2920 n and co-ordinate theserepresentations using the pooling operator 2950 to concatenate therepresentations across all images and use a final damage classifiermodel 2960 to make final predictions. In alternative embodiments, thepooling operator 2950 can use a pooling operation, e.g. a mean, maxoperation, with the representations produced by the visual damageclassifier 2940.

In alternative embodiments, the cropping is optional and the multi-imagemodel 2930 can be used with the original images 2910.

Referring now to FIG. 30, which show an alternative embodiment 3000 tothat shown in FIG. 29, the alternative embodiment using a graph neuralnetwork, and this alternative embodiment will now be described in moredetail.

Multiple images 3010 are input into the process 3000, each of which iscropped into multiple crops of each image 3020′, 3020″, 3020′″, 3020 n.Each of the multiple crops of each image 3020′, 3020″, 3020′″, 3020 nare then processed by a multi-image graph model 3030. In thisembodiment, the multi-image model 3030 comprises a visual damageclassifier 3040, then a graph convolutional neural network 3060, whichoutputs a final damaged/undamaged score 3070 for each part.

As in the previous embodiment, in this embodiment by cropping themultiple images 3010, the image resolution of the images are preservedin the cropped images 3020′, 3020″, 3020′″, 3020 n so that the fulldetail of the original resolution of the images can be processed by themulti-image model 3030 without needing to down-sample the images 3010.In alternative embodiments, instead of cropping the images 3010, themodel 3030 can be trained to work at the original resolution of theinput images without down-sampling.

By training and using a multi-image graph model 3030, all of the images3010 can be considered when making a prediction 3070, as the graphneural network 3060 treats each feature representation as a node in agraph. The model 3030 can thus also substantially overcome thelimitations in the input data only showing damage clearly in some imagesof the vehicle and not in others, as the model can have arbitraryconnections between nodes in the graph to represent this. Themulti-image model 3030 achieves this by the visual damage classifier3040 producing a feature representation for each of the images 3010and/or multiple cropped images 3020′, 3020″, 3020′″, 3020″ and thenusing the graph neural network 3060 to make final predictions.

Referring now to FIG. 31, there is shown a process using a languagemodel to extract information relevant to visual damage 3100, accordingto an embodiment, which will now be described in more detail.

To determine whether damage is consistent with the reasons for thedamage, one can compare the details of the accident/damage as reported(for example to an insurance company) and the determined damage to thevehicle.

To do this, in this embodiment, images of the damage to the vehicle 3110are processed by a per-part visual model 3120 to generate a visualrepresentation 3130. In this embodiment, there are multiple visualmodels 3120, each trained on a normalised part of a vehicle and trainedto determine if each part is classified as damaged or undamaged. Theoutput visual representation 3130 comprises classifications of damagedor undamaged for each normalised part of the vehicle based on the images3110. In some embodiments, the visual models 3120 can predict a richrepresentation (similar to or the same as that described in relation toother embodiments described herein), so for example can predict any orany combination of checks such as whether to repair or replace a part,the number of labour hours required, paint requirements, blendrequirement, etc. and/or can also predict other visual features such aslocation of damage and/or type of damage. Any or any combination ofthese predictions can be relevant to determine whether and/or correlatewith the vehicle was/being stationary or in movement when the damageoccurred.

In parallel, the order sheet 3140 or other text/data containinginformation relating to how the damage to the vehicle occurred (or isclaimed to have occurred) is input into a language model extractor 3150.The language model extractor 3150 is a natural language processing modeltrained to extract the circumstances of the damage 3160. Thecircumstances might include, for example, the point of impact, whetherthe vehicle was stationary or in motion, and the type of object withwhich the vehicle collided.

A classifier 3170 receives the output visual representation 3130comprising classifications of damaged or undamaged for each normalisedpart of the vehicle and the circumstances of the damage 3160 andclassifies whether the damage to each normalised part is consistent ornot with the extracted circumstances 3160. For example, the location ofdamage may be inconsistent with the circumstances (e.g. it would beinconsistent to have damage on the front bumper and roof where thedamage is not severe enough to indicate the vehicle rolled over) or thetype of damage may be inconsistent (e.g. it would be inconsistent tohave impact damage on a door and rust on a boot lid both being repaireddue to an accident) or the severity of the damage may be inconsistent(e.g. it would be inconsistent to have scratches on the front bumper anda bigger dent on the bonnet, indicating the damage occurred at differenttimes).

To train these models, the language model weights might be frozen whilethe visual model trains or both networks are trained jointly,end-to-end.

Referring now to FIG. 32, an alternative embodiment to that of FIG. 31is presented in the situation where structure accident data isavailable, which will now be described in more detail.

As mentioned above, in this embodiment the accident data in the ordersheet 3220 is stored (for example by the insurer computer system when aninsurance claim is made) in a structured format that allows for directprocessing of the structured data (i.e. no natural language processingis required to process the order sheet to extract the pertinentinformation regarding the circumstances of the damage).

The photos 3210 of the damaged vehicle and the order sheet 3220including the details of the circumstances of the damage are provided tothe model 3200. The circumstances of the damage to the vehicle areextracted from the structured data in an accident data extraction andinjection step 3230. The circumstances might include, for example, thepoint of impact, whether the vehicle was stationary or in motion, andthe type of object with which the vehicle collided.

The photos 3210 are processed by a visual damage classifier 3240, intowhich is also injected the circumstances of the damage to the vehicle,to predict whether there is damage to each part of the vehicle. A visualrepresentation 3250 is output by the classifier 3240 including theclassification of which parts are damaged or undamaged along with thecircumstances of the damage. The representation can include the locationof damage, the type of damage and the severity of damage.

The representation is provided to a further classifier 3260 whichoutputs a prediction of whether the damage to each part is consistentwith the circumstances extracted from the order sheet 3220. For example,the location of damage may be inconsistent with the circumstances (e.g.it would be inconsistent to have damage on the front bumper and roofwhere the damage is not severe enough to indicate the vehicle rolledover) or the type of damage may be inconsistent (e.g. it would beinconsistent to have impact damage on a door and rust on a boot lid bothbeing repaired due to an accident) or the severity of the damage may beinconsistent (e.g. it would be inconsistent to have scratches on thefront bumper and a bigger dent on the bonnet, indicating the damageoccurred at different times).

To train these models, the language model weights might be frozen whilethe visual model trains or both networks are trained jointly,end-to-end.

Referring now to FIG. 33, there is shown a process 3300 for predictingdamage to grouped auxiliary parts associated with a normalised panelaccording to an embodiment, which will now be described in more detail.

Auxiliary parts are parts of a vehicle such as any or any combinationof: cameras, headlights, sensors, minor grilles, numberplates, emblems,fuel lids, brackets and such like. Some of these auxiliary parts tend tobe difficult to determine accurately in images of a vehicle, and damageto these auxiliary parts can also tent to be difficult to determine fromimages of a vehicle.

Photos 3310 of the damage to the vehicle are input into the process 3300and provided to an intermediary visual classifier 3320. The intermediaryvisual classifier 3320 comprises a plurality of per-part classifiers,each trained to determine a classification of damage to a normalisedpart of the vehicle for each of the input images 3310 and how severe thedamage is. For example, the severity can range from 1 (scratches) to 5(all areas affected).

In other embodiments, the intermediary visual classifier is trained topredict the location of the damage within the normalised part (forexample: left/centre/right).

The output classifications of the intermediary visual classifier 3320are provided to a multi-task learned classifier 3330, which is trainedto simultaneously predict damage to multiple auxiliary parts of thevehicle per normalised part, based on the damage classifications to thenormalised parts of the vehicle for the photos 3310.

As auxiliary parts, typically multiple auxiliary parts, can beassociated with specific normalised parts (e.g. for a front bumper,auxiliary parts can include minor grilles, fog lamps, numberplates,emblems, parking sensors, impact bars, etc), a trained multi-taskclassifier can learn common features between all auxiliary parts on eachnormalised part/panel/region. This avoids training per-part models forauxiliary parts, which can be harder to train as not all vehicles havesome auxiliary parts thus training data tends to be unbalanced orminimal.

The multi-task learned classifier 3330 can take the informationdetermined by the visual classifier 3320 and use this to determine thelikelihood of damage to one or more of the plurality of auxiliary partsassociated with each normalised part based on for example the severityof the damage to each normalised part and whether the damage is presentin a region of the normalised part in which the auxiliary part(s) arelocated.

In alternative embodiments, the visual classifier 3320 can perform asegmentation of the damage to each normalised part to identify moreprecisely where the damage on the vehicle is located, providing richerdata for the multi-task learned classifier 3330 to predict whetherdamage has occurred to any auxiliary parts using this segmentation data.Further, in other embodiments, the visual classifier 3320 can perform aclassification of the category of damage (e.g. scratch, dent, tear,rust, etc). to increase the richness of the data for the multi-tasklearned classifier 3330 to predict whether damage has occurred to anyauxiliary parts.

Finally, an output classification is output 3340 by the multi-tasklearned classifier 3330 indicating whether any auxiliary parts aredetermined to be damaged. In some embodiments, the multi-task learnedclassifier 3330 takes just the output of the intermediary visualclassifier 3320 as an input. In other embodiments, the multi-tasklearned classifier 3330 takes both the input images 3310 and the outputof the intermediary visual classifier 3320 as inputs, for example theoutput of the intermediary visual classifier 3320 might comprise one ormore segmentation masks of the images 3310, thus the images will beneeded in addition to the segmentation masks to determine theclassifications 3340).

In other embodiments, the intermediary visual classifier 3300 may not berequired and just a multi-task classifier 3330 is used.

Referring now to FIG. 34, an example multi-task architecture that can beused in the embodiment shown in FIG. 33 is shown, and will now bedescribed in more detail.

The multi-task architecture shown has both shared layers 3460, 3430,3480 and task specific layers 3410, 3420, 3490, 3440, 3450, 3470. Inthis example, there are three tasks to be performed by the model, tasksA, B and C. Task A has some task-specific layers 3440, 3410 whilesimilarly task B has some task-specific layers 3450, 3420 and task C hassome task-specific layers 3470, 3490.

In the example embodiment of FIG. 33, each of tasks A, B and C mightrelate to one of three auxiliary parts on one of the normalised parts.

Referring now to FIG. 35, there is shown a process 3500 for jointlytraining across variably sized datasets, according to an embodiment,which will now be described in more detail.

In some cases, models 3530 are required to be used with differentdatasets 3510, 3520, for example for datasets from different geographies3510, 3520. Further, where the different datasets 3510, 3520 aresignificantly different sizes relative to each other, or when onedataset is too small to train a model 3530 to output accurate results,it can prove hard to train a sufficiently accurate model 3530 for bothdatasets. One option is to train the model 3530 on the larger dataset3520 and fine tune it on the smaller dataset 3510 in order to be able tooutput sufficiently accurate predictions for the smaller dataset 3510,but even this might fail if the smaller dataset 3510 is too dissimilarin size relative to the larger dataset 3520.

In this embodiment, a model 3530 is trained jointly on both datasets3510, 3520 to evaluate the extent of the damage per normalised part, andthen to predict a probability as to whether a panel should be replacedand the number of hours it would take to replace it. A multi-learningapproach similar to that presented in other embodiments can be used totrain the model 3530. The predictions can then be adapted for eachdataset/geography to output final predictions 3530 per dataset 3510,3520.

Further, in this embodiment, a domain confusion loss 3540 is injectedinto the model 3530 mid-way through the network to ensure that thenetwork can't differentiate between data from the first dataset 3510 andthe second dataset 3520, thus training the model 3530 to be generalisedacross both datasets 3510, 3520 (and therefore, for example, to workacross both geographies). In alternative embodiments, no domainconfusion loss 3540 is used.

Referring now to FIG. 36, an alternative approach to working withmultiple different sized datasets 3600 is presented according to anotherembodiment, which will now be described in more detail.

In this example, there are two datasets 3610, 3620—the first geographydataset 3610 is much smaller than the second geography dataset 3620. Ifthe most relevant data from the larger dataset 3620 can be identified,then it can be used to jointly train a model 3650 on the first dataset3610 and a subset of the second dataset 3620. To identify the mostrelevant data from the larger dataset 3620, the most similar data pointsfrom the larger dataset 3620 are selected (compared to the smallerdataset 3610) and used to populate a reduced second geography dataset3640. In this embodiment, classifiers trained on the larger dataset 3620are used to determine which data points are similar to the data in thesmaller dataset 3610. This can be done by creating visual damage vectorsfor all the data in both the larger and smaller datasets 3610, 3620 andthen finding the data points in the vectors/data points created for thelarger dataset 3620 that are most similar/close to the vectors/datapoints created for the smaller dataset 3610. To determine how “close”the vectors/data points are, in terms of visual damage vectors, a simpledistance measure can be used (for example Euclidian, Mean Squared, etc.)or a simple classifier can be trained (for example using linearregression or a support vector machine) to separate the two datasets anddetermine which points in the larger dataset don't get separated fromthe smaller dataset. If even a small amount of the larger dataset 3620is selected 3640 to augment the first dataset 3610, then it is possiblethat a substantially improved model 3660 can be generated followingjoint training across the first dataset 3610 and second dataset 3640.

Referring to FIG. 37, there is shown a example domain confusion lossarchitecture 3700 that can be used with other embodiments, which willnow be described in more detail.

Two networks 3720, 3790 are shown, one network 3730 for labelled images3760 and one network 3790 for unlabelled images 3780. Each network has anumber of convolutional layers 3730, 3710, 3792, 3791 and fullyconnected layers 3714, 3713, 3712, 3711, 3795, 3794, 3793, 3792, 3770and have a domain loss 3740 injected in between layers 3770, 3794, 3714,3712. Additionally, there is a classification loss 3750 injected intoone of the networks 3720. Additional, there is an adaption layer 3713,3795 in each network 3720, 3790. In this example, the two domains beingconsidered are labelled and unlabelled images, and the goal of thedomain confusion loss 3740 is to make sure that at the fc8 layer 3714,3770 the network can't differentiate between data that came from thelabelled domain 3760 and data that came from the unlabelled domain 3780.It does this by having a small classifier on top of the fc8 layer 3714,3770 which tried to predict whether a particular image came from thelabelled or unlabelled domains. If the small classifier predictscorrectly, it penalizes the fc8 layer 3714, 3770 in the network. In thisway, while training, the network will learn that in the fc8 layer 3714,3770 it should not be possible to differentiate data from the twodomains 3760, 3780, thus leading to better generalisation across the twodomains 3760, 3780.

It should be understood by the reader that the cropping and/orsegmentation techniques described in some embodiments herein can beadapted for use in other embodiments and/or aspects described herein,for example in the determination of whether a part is damaged orundamaged and/or the determination of damage to auxiliary parts. Itfollows that these techniques are thereby disclosed for use in relationwith any aspects/embodiments herein.

It should also be understood by the reader that the multi-image learningapproaches described in embodiments and/or aspects herein can beincorporated into other aspects/embodiment to substantially improveaccuracy for all classification or models across multiple images and/orother input data. It follows that these approaches are thereby disclosedfor use in relation with any aspects/embodiments herein.

The above described embodiments will be able to be implemented using avariety of software and hardware, all within the intended scope of theaspects and embodiments set out herein. For example, the software may bewritten and deployed in a variety of ways using a variety of languagesand implementations. The hardware that may be used to deploy thesoftware and methods/systems described herein according to anyembodiment and/or aspect may be a variety of different types andconfigurations of hardware and hardware architecture including the useof any or any combination of: handheld devices, mobile ‘phones, smart‘phones, cameras, imaging devices, tablet computers, desktop computers;laptop computers; virtual desktops; servers; distributed servers;distributed computing arrangements; and cloud systems.

By models and neural networks, the use of machine learning andartificial intelligence techniques are intended to be referred to,including the use of deep learning techniques and convolutional andother neural networks.

Machine learning is the field of study where a computer or computerslearn to perform classes of tasks using the feedback generated from theexperience or data gathered that the machine learning process acquiresduring computer performance of those tasks.

Typically, machine learning can be broadly classed as supervised andunsupervised approaches, although there are particular approaches suchas reinforcement learning and semi-supervised learning which havespecial rules, techniques and/or approaches. Supervised machine learningis concerned with a computer learning one or more rules or functions tomap between example inputs and desired outputs as predetermined by anoperator or programmer, usually where a data set containing the inputsis labelled.

Unsupervised learning is concerned with determining a structure forinput data, for example when performing pattern recognition, andtypically uses unlabelled data sets. Reinforcement learning is concernedwith enabling a computer or computers to interact with a dynamicenvironment, for example when playing a game or driving a vehicle.

Various hybrids of these categories are possible, such as“semi-supervised” machine learning where a training data set has onlybeen partially labelled. For unsupervised machine learning, there is arange of possible applications such as, for example, the application ofcomputer vision techniques to image processing or video enhancement.Unsupervised machine learning is typically applied to solve problemswhere an unknown data structure might be present in the data. As thedata is unlabelled, the machine learning process is required to operateto identify implicit relationships between the data for example byderiving a clustering metric based on internally derived information.For example, an unsupervised learning technique can be used to reducethe dimensionality of a data set and attempt to identify and modelrelationships between clusters in the data set, and can for examplegenerate measures of cluster membership or identify hubs or nodes in orbetween clusters (for example using a technique referred to as weightedcorrelation network analysis, which can be applied to high-dimensionaldata sets, or using k-means clustering to cluster data by a measure ofthe Euclidean distance between each datum).

Semi-supervised learning is typically applied to solve problems wherethere is a partially labelled data set, for example where only a subsetof the data is labelled. Semi-supervised machine learning makes use ofexternally provided labels and objective functions as well as anyimplicit data relationships. When initially configuring a machinelearning system, particularly when using a supervised machine learningapproach, the machine learning algorithm can be provided with sometraining data or a set of training examples, in which each example istypically a pair of an input signal/vector and a desired output value,label (or classification) or signal. The machine learning algorithmanalyses the training data and produces a generalised function that canbe used with unseen data sets to produce desired output values orsignals for the unseen input vectors/signals. The user needs to decidewhat type of data is to be used as the training data, and to prepare arepresentative real-world set of data. The user must however take careto ensure that the training data contains enough information toaccurately predict desired output values without providing too manyfeatures (which can result in too many dimensions being considered bythe machine learning process during training and could also mean thatthe machine learning process does not converge to good solutions for allor specific examples). The user must also determine the desiredstructure of the learned or generalised function, for example whether touse support vector machines or decision trees.

The use of unsupervised or semi-supervised machine learning approachesare sometimes used when labelled data is not readily available, or wherethe system generates new labelled data from unknown data given someinitial seed labels.

Machine learning may be performed through the use of one or more of: anon-linear hierarchical algorithm; neural network; convolutional neuralnetwork; recurrent neural network; long short-term memory network;multi-dimensional convolutional network; a memory network; fullyconvolutional network or a gated recurrent network allows a flexibleapproach when generating the predicted block of visual data. The use ofan algorithm with a memory unit such as a long short-term memory network(LSTM), a memory network or a gated recurrent network can keep the stateof the predicted blocks from motion compensation processes performed onthe same original input frame. The use of these networks can improvecomputational efficiency and also improve temporal consistency in themotion compensation process across a number of frames, as the algorithmmaintains some sort of state or memory of the changes in motion. Thiscan additionally result in a reduction of error rates.

Developing a machine learning system typically consists of two stages:(1) training and (2) production. During training, the parameters of themachine learning model are iteratively changed to optimise a particularlearning objective, known as the objective function or the loss. Oncethe model is trained, it can be used in production, where the modeltakes in an input and produces an output using the trained parameters.

During the training stage of neural networks, verified inputs areprovided, and hence it is possible to compare the neural network'scalculated output to then the correct the network is need be. An errorterm or loss function for each node in neural network can beestablished, and the weights adjusted, so that future outputs are closerto an expected result. Backpropagation techniques can also be used inthe training schedule for the or each neural network.

The model can be trained using backpropagation and forward pass throughthe network. The loss function is an objective that can be minimised, itis a measurement between the target value and the model's output.

The cross-entropy loss may be used. The cross-entropy loss is defined as

$L_{CE} = {- {\sum\limits_{c = 1}^{C}{y*{\log(s)}}}}$

where C is the number of classes, y∈{0,1} is the binary indicator forclass c, and s is the score for class C.

In the multitask learning setting, the loss will consist of multipleparts. A loss term for each task.

L(x)=+λ₁ L ₁+λ₂ L ₂

where L₁, L₂ are the loss terms for two different tasks and λ₁, λ₂ areweighting terms.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect may be applied to other aspects, in anyappropriate combination. In particular, method aspects may be applied tosystem aspects, and vice versa. Furthermore, any, some, and/or allfeatures in one aspect can be applied to any, some and/or all featuresin any other aspect, in any appropriate combination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects can be implementedand/or supplied and/or used independently.

1-16. (canceled)
 17. A method, comprising: receiving one or more imagesof a damaged vehicle; determining one or more classifications for one ormore parts of the damaged vehicle based on at least the one or moreimages; generating one or more damage vectors based on the one or moreclassifications; and determining one or more repair operations for thedamaged vehicle based on at least the one or more damage vectors. 18.The method of claim 17, wherein the one or more repair operationscomprise replacing or repairing a damaged part of the damaged vehicle.19. The method of claim 17, further comprising: determining a severityof damage to one or more parts of the damaged vehicle based on at leastthe one or more damage vectors.
 20. The method of claim 17, furthercomprising: predicting a repair cost corresponding to the damagedvehicle based on at least the one or damage vectors.
 21. The method ofclaim 17, further comprising: determining one or more paintingoperations for the damaged vehicle based on at least the one or moredamage vectors.
 22. The method of claim 17, further comprising:determining a labor time parameter corresponding to the damaged vehiclebased on at least the one or more repair operations, wherein the one ormore repair operations comprise repairing at least one part of thedamaged vehicle.
 23. The method of claim 17, further comprising:generating a damage vector for the damaged vehicle based on the one ormore damage vectors, wherein each of the one or more damaged vectorscorrespond to a different normalized part of the damaged vehicle. 24.One or more processors configured to perform operations comprising:receiving one or more images of a damaged vehicle; determining one ormore classifications for one or more parts of the damaged vehicle basedon at least the one or more images; generating one or more damagevectors based on the one or more classifications; and determining one ormore repair operations for the damaged vehicle based on at least the oneor more damage vectors.
 25. The one or more processors of claim 24,wherein the one or more repair operations comprise replacing a damagedpart of the damaged vehicle.
 26. The one or more processors of claim 24,further comprising: determining a severity of damage to one or moreparts of the damaged vehicle based on at least the one or more damagevectors.
 27. The one or more processors of claim 24, further comprising:predicting a repair cost corresponding to the damaged vehicle based onat least the one or damage vectors.
 28. The one or more processors ofclaim 24, further comprising: determining one or more paintingoperations for the damaged vehicle based on at least the one or moredamage vectors.
 29. The one or more processors of claim 24, furthercomprising: determining a labor time parameter corresponding to thedamaged vehicle based on at least the one or more damaged vectors. 30.The one or more processors of claim 24, further comprising: generating adamage vector for the damaged vehicle based on the one or more damagevectors, wherein each of the one or more damaged vectors correspond to adifferent normalized part of the damaged vehicle.
 31. A non-transitorycomputer-readable storage medium storing a set of instructions that isexecutable by one or more processors, the set of instructions, whenexecuted by the one or more processors, causing the one or moreprocessors to perform operations, comprising: receiving one or moreimages of a damaged vehicle; determining one or more classifications forone or more parts of the damaged vehicle based on at least the one ormore images; generating one or more damage vectors based on the one ormore classifications; and determining one or more repair operations forthe damaged vehicle based on at least the one or more damage vectors.32. The non-transitory computer-readable storage medium of claim 31,wherein the one or more repair operations comprise replacing orrepairing a damaged part of the damaged vehicle.
 33. The non-transitorycomputer-readable storage medium of claim 31, the operations furthercomprising: predicting a repair cost corresponding to the damagedvehicle based on at least the one or damage vectors.
 34. Thenon-transitory computer-readable storage medium of claim 31, theoperations further comprising: determining one or more paintingoperations for the damaged vehicle based on at least the one or moredamage vectors.
 35. The non-transitory computer-readable storage mediumof claim 31, the operations further comprising: determining a labor timeparameter corresponding to the damaged vehicle based on at least the oneor more repair operations, wherein the one or more repair operationscomprise repairing at least one part of the damaged vehicle.
 36. Thenon-transitory computer-readable storage medium of claim 31, theoperations further comprising: generating a damage vector for thedamaged vehicle based on the one or more damage vectors, wherein each ofthe one or more damaged vectors correspond to a different normalizedpart of the damaged vehicle.